What Codex auto review actually catches

The one-line version

Codex auto review runs a model over a diff and leaves comments. That is the whole idea. It happens either from the CLI, where you ask for a review of your working changes before you push, or on a pull request, where it comments like any other reviewer. The output is a list of findings tied to lines.

Teams get confused because they expect it to behave like a linter. It does not. A linter is deterministic. Run it twice on the same file and you get the same answer. Run a model review twice and you get overlapping but different sets of comments. Plan for that, and stop treating a clean pass as proof of anything.

Where it earns its keep

In the teams we train, the findings that survive scrutiny cluster in a narrow band:

• Error paths that were written but never wired up. A caught exception that logs and continues when it should return.
• Off-by-one and boundary conditions in loops and slicing.
• Values that go into a string interpolation and end up in SQL, a shell command, or a URL.
• A function signature changed in one place and left stale at two call sites the author forgot.
• Tests that assert the mock rather than the behaviour.

That last one is the sleeper. A reviewer skimming a 900-line diff at 5pm will nod at a test file. The model reads every line of it at the same attention level as the first line, which is the actual advantage. Not intelligence. Stamina.

It is also decent at the tedious cross-file check. If a diff renames a config key, it will usually find the deployment file three directories away that still uses the old name. A human reviewer looking at a diff view often cannot see that file at all.

Where Codex auto review is weak

It reviews the diff, so it cannot tell you the diff should not exist. It will happily polish an approach that a senior would have rejected outright. It also has no memory of the last four incidents your team had, no sense of which module is load-bearing, and no way of knowing that the config flag being added duplicates one added six months ago under a different name.

Second failure mode: confident nitpicking. You get comments about naming and structure that sound authoritative and are just preference. If you leave those unmanaged, engineers learn to scroll past every comment, and then the real finding gets scrolled past too.

Third: large diffs. The signal degrades as the change grows. A 40-file refactor produces vague findings. That is a hint about your pull request size as much as about the tool. If the reviewer cannot hold the change, neither can the person approving it.

Wiring it in without wrecking the review culture

Two rules do most of the work.

First, run it before the human. The point is that a person never spends attention on a missing null check. Ask for the review locally on your own diff, fix what is real, then open the pull request:

git diff main... | review the changes for correctness bugs only

Second, constrain the scope in the repo instructions. Put a short block in AGENTS.md stating what a review comment is allowed to be about. Something like: correctness, security, data loss, and public API changes. Style is the formatter's job. Naming is the author's job. We have seen review comment volume drop by more than half after that one edit, with the useful findings intact.

Then track one number for a month: how many auto review comments led to a code change. If it is under one in five, your prompt or your scope is wrong, not the model. Keep it advisory rather than blocking, since a gate whose output varies between runs teaches people to bypass gates.

What to do next

Pick your last three merged pull requests that caused a bug. Run the review against those diffs and see whether it would have caught them. That test takes an hour and tells you more than any vendor page, including this one. If it catches two of three, roll it out and write the scope rules first.

If you want help putting this into practice, talk to us.