Simon Willison on Coding Agent Review

Simon Willison’s August 22, 2026 post, “More than just code review,” is a short engineering note about how developers should review work produced by coding agents. It deals with a practical question: how do you trust an AI-made change without pretending that reading every generated line is always enough? The useful answer is simple: the core skill is not code review as a ritual, but confident instruction plus confident verification. Agentic coding is the practice of delegating software changes to an AI coding agent while the developer stays responsible for the goal, boundaries, and proof that the change is correct.

Read the post as a shift in review work

Willison’s point lands because it names a thing many developers are already feeling. When a coding agent produces a small patch, line-by-line review still works. When it touches a migration, updates tests, edits fixtures, and refactors a helper along the way, “just read the diff” starts to feel like staring at a map instead of driving the route.

The post does not say humans should stop reading code. It says reading code is one verification method, not the whole job. That matters for Codex, OpenAI’s coding agent, because a useful Codex session often leaves you with a working branch, a test trail, and a set of design choices to inspect.

The trap is letting the agent’s confidence become your confidence. A tidy diff can still encode the wrong product rule. A green test can still miss the path the customer actually uses.

A concrete example: ask a coding agent to “add rate limiting to login.” The change might add middleware, tests, and configuration. The human question is not only “is this code neat?” It is “does this enforce the intended limit, in the intended place, with the intended failure mode, without locking out trusted traffic?”

Verify the behavior, not the transcript

The most useful part of Willison’s note is the reminder that validation has always been bigger than eyeballing code. Good engineers already verify with tests, type checks, logs, small repros, staging behavior, migration dry runs, and production-like data. Coding agents make that old truth harder to ignore.

For a Codex workflow, the review target should be the resulting branch, not the chat transcript. The transcript may explain intent, but the repo is the artifact. Start with git diff --stat, inspect the risky files, run the narrow tests, then run the broader checks that would have caught a human mistake.

Here is a small Codex CLI verification loop you can adapt:

git switch -c agent/login-rate-limit
codex exec "Add per-IP rate limiting to login. Keep behavior documented in tests."

git diff --stat
git diff -- src/auth tests/auth
npm test -- auth
npm run typecheck

The trap is replaying the whole agent conversation as if it were evidence. It is useful context, but it is not proof. Proof lives in commands you can rerun and behavior you can observe.

This is also where code review guardrails become less bureaucratic and more practical. A guardrail can be as small as “database migrations must include rollback notes” or “auth changes require a failing test first.” We track broader patterns under the related training topic, but the useful unit is still one branch and one verified change.

Put the rule where the agent will see it

A coding agent can only follow constraints it can access. If a repo has sharp local rules, put them in AGENTS.md near the code they govern. Keep the rule short enough that a person would actually read it too.

For example, in a service with sensitive login behavior:

# AGENTS.md

## Auth changes

When changing files under `src/auth/`:

- Add or update tests in `tests/auth/` for the user-visible behavior.
- Preserve existing lockout and audit-log behavior unless the task says otherwise.
- Run `npm test -- auth` and `npm run typecheck` before handoff.
- In the final note, name the risk you checked and the command output you used.

This is not magic. It is a local contract. It gives Codex a review shape before it starts editing, and it gives the human a receipt to compare against after the change.

The trap is stuffing AGENTS.md with every preference the team has ever discussed. Durable rules belong there. Task-specific wishes belong in the prompt. If everything is mandatory, nothing feels mandatory.

Keep tool access boring on purpose

Willison’s argument also points at a quiet boundary question: what can the agent touch while making the change? Model Context Protocol (MCP) is an open protocol for connecting AI applications to tools and data, and it is becoming the common way to give agents access to systems like GitHub, docs, issue trackers, and databases.

For review work, boring access is good access. Start with read-only context where possible. Let the agent read the issue, inspect docs, and query a replica before it can mutate external state.

A light MCP boundary note can be enough for a repo experiment:

## MCP boundary for this task

- GitHub issues: read only.
- Docs search: read only.
- Database: read-only staging replica only.
- Writes are limited to the working tree on this branch.

The trap is treating tool access as a convenience toggle. More access can improve agent performance, but it also expands what you must verify. If the agent can change code and update an issue and write to a database, your review surface just got much larger.

For a related look at agents working through multiple branches at once, see Zuse Runs 20 Linear Issues in Worktrees. The same lesson applies: parallelism is only useful when the receipts are easy to inspect.

Try it safely on one real change

Use Willison’s note as a prompt to change one habit, not as a reason to redesign your whole process. Pick a small but non-trivial change where behavior matters: a validation rule, an error message, a query filter, or a permission check.

Copy this checklist into the task or PR description:

## Coding agent review receipt

Change requested:
- What user-visible behavior should change?

Boundaries:
- Which files or systems should not be touched?
- Which MCP tools, if any, are read-only?

Verification:
- What narrow test proves the behavior?
- What broader command catches integration mistakes?
- What manual check would reveal the most likely wrong implementation?

Reviewer notes:
- Which diff areas are high risk?
- Which command outputs did you rely on?
- What did you not verify?

The point is not to avoid reading the diff. Read the diff where risk is concentrated. But make the agent hand you something better than “done”: a small chain of evidence you can rerun.

The limitation is real. Some changes are hard to verify mechanically, especially UI taste, ambiguous product behavior, security-sensitive design, and performance under production load. In those cases, the receipt should say what remains uncertain instead of pretending the agent proved everything.

Common questions

• Does this mean AI code review is obsolete?

  No. AI code review still matters, but it should sit beside behavioral verification. Willison’s point is that reading every line has never been the only way to validate software; tests, focused diffs, repro steps, and runtime checks often catch mistakes that pure inspection misses.

• What should I review first in a Codex-generated branch?

  Review the intended behavior first, then the risky diff. Start with the task statement, git diff --stat, the files that cross trust boundaries, and the verification commands the agent claims to have run. If the branch changes auth, billing, permissions, migrations, or data deletion, slow down.

• Where does MCP fit into coding agent review?

  MCP fits at the tool boundary. If an agent used an MCP server to read issues, query docs, or inspect a database, the review should include what access it had and whether any writes were possible. A read-only MCP boundary is easier to reason about than broad tool access.

• Is an AGENTS.md file enough to keep an agent safe?

  No. AGENTS.md helps the agent follow repo rules, but it is not a safety system by itself. Treat it as instruction, then verify the result with tests, diffs, logs, and human judgment. The file is strongest when it names local constraints and required checks.

• How do I know when line-by-line review is still required?

  Use line-by-line review when the change is small, security-sensitive, or hard to test. Generated code that touches authentication, authorization, cryptography, data loss, migrations, or public APIs deserves close inspection. The better rule is not “always read everything”; it is “match review depth to risk.”

Best ways to use this research

• Best for: Developers using Codex or another coding agent who already get useful diffs, but feel unsure how much review is enough.
• Best first artifact: A short AGENTS.md verification rule for one risky directory, such as auth, billing, permissions, or migrations.
• Best comparison angle: Compare “read every generated line” with “prove the requested behavior changed and the risky boundaries held.” The second usually gives better evidence.
• Best next experiment: Run one agent task with a required review receipt, then decide whether the receipt made the PR easier or harder to trust.

Further reading

• Simon Willison — source
• Model Context Protocol — specification
• Codex — Agent
• OpenAI Codex CLI — official repository

Next step

Pick one Codex-generated change this week and require a review receipt before you read the full diff. If the receipt does not help, the task was probably too vague or the verification commands were too weak.

One methodology lens

One useful way to read this through our methodology is the Plan step: delegate first-pass decomposition and dependency mapping, review the sequencing and assumptions, and keep ownership of scope and priorities. If that split is still fuzzy, the workflow usually is too.