Autoreview in Codex and how teams get value from it

What autoreview is good at

Automated review reads a diff and reports problems. Run it across a few hundred real pull requests and a pattern emerges. It is reliable on the mechanical class of bug: an error that gets caught and dropped, a resource never closed, an early return that skips cleanup, a boolean inverted from the name of the function it lives in, a new call to something that was deprecated last quarter.

That is not a small list. Those are the bugs that survive human review, because a human reading a 300 line diff at 5pm skims exactly that kind of line.

What autoreview in Codex misses

It does not know your product. A pull request that computes VAT correctly but applies it to the wrong customer segment looks fine to a reviewer that has never seen your pricing rules. It does not know your incident history. It does not know that the module it just praised is scheduled for deletion.

It also degrades on large diffs. The relationship is not subtle: as the change grows, the comments get more generic and the real findings thin out. If your team routinely opens 2000 line pull requests, fix that first. The review will improve as a side effect, and so will the human reviews.

Tuning it so people keep reading it

The measure that matters is what fraction of comments produce an edit. Below roughly a third, engineers stop reading, and after that the tool is worse than nothing because it creates a false sense of coverage. To get there:

• Write the standards down. Put your real conventions in AGENTS.md in plain sentences, including the ones you argue about in review.
• Exclude generated code, lockfiles, snapshots, and vendored directories. Nobody wants review comments on a migration your ORM wrote.
• Skip bot-authored dependency pull requests entirely.
• Tell it what not to raise. Style opinions belong to your formatter, not your reviewer.
• Keep it advisory. A required check that blocks merge on a model's opinion turns into a rubber stamp within a month.

Try it locally on a branch before wiring it into CI. Something like git diff main... | wc -l first, so you know how big the thing you are asking it to read actually is.

A failure worth knowing about

A team we trained had review enabled on every push. One engineer force-pushed eleven times while rebasing. Eleven full review passes on the same diff, eleven near-identical comment threads, and the actual reviewer gave up and approved without reading. The fix was one line of CI config restricting review to pull request open and to non-draft branches. Nothing about the model changed.

What to do next

Open your last month of pull requests and count, roughly, how many automated review comments led to a code change. If it is under a third, do not tune the prompt yet. Exclude generated files and bots, cap diff size in your team norms, and write your actual conventions down. Then count again in a month. The number is the only thing that tells you whether this is helping.

If you want help putting this into practice, talk to us.