Codex-auto-review: what it catches and misses

Auto-review catches syntax drift and misses permission drift. We turned it on, the merge queue sped up, and the audit a week later showed exactly that split. Codex-auto-review is the practice of letting Codex, OpenAI's coding agent, review diffs automatically while a human keeps the merge decision. It is worth running, but the green checkmark guarantees less than it looks like, and the gap is always around scopes and permissions.

So treat auto-review as a filter, not a verdict. It clears the boring stuff so your eyes land on the parts that actually need a human: who can touch what, and why.

Trust the transcript, not the pass rate

The first sprint, we made a quiet mistake. We audited only the diffs auto-review flagged and never the ones it waved through. The waved-through pile is where every permission problem was hiding.

That is survivorship bias in plain clothes. The diffs the tool catches leave a report behind. The ones it misses leave nothing, so the evidence pile flatters the tool by design. A clean dashboard told us things were fine while a connector quietly gained scopes it never should have had.

The fix is cheap. Ask every run to end with a transcript snippet a reviewer can replay: the actual commands, the actual output. Confidence should track that snippet, not a screenshot and not the pass count.

Four gaps to guard before you widen autonomy

These are the four places auto-review gets quiet when you most want it loud. Each one has a small written contract that closes it.

Verification bypass. Exec shortcuts skip tests, a regression slips back in, and the automated pass makes the shortcut feel safe. Require a transcript snippet showing tests ran after the code was generated. Now a green merge means the ritual happened, not that someone felt good about it.

Headless versus browser drift. Chrome workflows diverge from CLI habits, so the reviewer sees two versions of the truth and the demo contradicts CI. Write the staging URLs and credential boundaries down next to any browser task.

Model mismatch. People swap models casually, and different models carry different risk appetite. Pin the default model and the escalation rule inside AGENTS.md so a lead can still reason about the blast radius of a change.

MCP privilege creep. Connectors accumulate, each server widens what an agent can reach, and this is the permission drift the trial actually exposed. Keep a Markdown roster of connectors checked into the repo root, grounded in the MCP specification.

Here is the snippet we keep at the bottom of AGENTS.md. Paste it, adjust the wording, ship it.

# AGENTS.md verification snippet

- Every Codex CLI run ends with the transcript snippet reviewers can replay.
- Pair browser evidence with the project's normal CLI checks before merge.
- If MCP servers are enabled, list allowed actions beside each connector name.

Keep four gates on every merge

When a model is the reviewer, the human questions do not change, they just get more deliberate. These four cover most of what auto-review cannot prove on its own.

And the short scope receipt a person still signs off on:

• Red-folder paths received explicit human acknowledgement.
• Scopes in the PR body match the folders in the diff.
• Primary-doc links were smoke-checked after publishing edits.
• MCP connectors mentioned (if any) list owners.

If you want the tool's side of the loop, the quickstart, the Codex CLI docs, the features page, and the slash commands reference cover it, and openai/skills shows where packaged behaviors are heading. The architecture judgement stays with the humans either way. For the full practice area, see Codex CLI workflows.

Common questions

• What is codex-auto-review?

  Codex-auto-review is the practice of letting Codex review diffs automatically while a human keeps the merge decision. In our trials it caught syntax drift but missed permission drift. That is why every automated pass still needs a transcript snippet showing tests ran after the code was generated, pasted somewhere a reviewer can replay it.

• Which model should codex-auto-review runs use?

  Pin one default model and one escalation rule inside AGENTS.md instead of letting people swap casually. Different models carry different risk appetite, so an unpinned model makes review expectations wobble. With it pinned, a lead can reason about the blast radius of a given merge instead of guessing.

• Does auto-review replace human code review?

  No. Auto-review speeds up execution while architecture judgement and ownership stay with people. The gates keep that line concrete: a reviewer still asks which commands prove the regression guards and whether the PR body lists scopes alongside the verification transcript. The model filters, the person decides.

• What evidence should a Codex auto-review run leave behind?

  A transcript snippet showing tests ran after the code was generated, pasted where reviewers can replay it. The scope receipt adds three checks on top: red-folder paths get explicit human acknowledgement, PR-body scopes match the folders in the diff, and any MCP connectors mentioned list their owners.

Start with the connector roster

If you only do one thing this week, write the connector roster, because permission drift is the failure auto-review demonstrably missed. To build the habit on your team, see how we run this in training.