Harness Design for Coding Agents, Measured

An empirical study of harness design for coding agents is the arXiv paper, a September 2026 research article about how coding-agent harnesses affect benchmark results, not just how smart the underlying model is. It studies a practical question: when an agent succeeds or fails at a coding task, how much of that result comes from planning, tools, shell access, and orchestration around the model? My takeaway is simple: agentic coding results are measured through a harness, so Codex, OpenAI's coding agent, users should review the harness boundary before treating a benchmark result as a model verdict. A coding-agent harness is the wrapper that gives a model instructions, tools, state, commands, feedback, and stopping rules while it works on code.

Read it as a harness paper, not a model race

The useful move in the paper is that it shifts attention away from the model name and toward the operating conditions around the model. That matters because two agents can use the same base model and still behave differently if one gets better planning support, cleaner tool APIs, or a safer path through the shell.

The paper reports that planning can improve success at extra cost for weaker models. For stronger models, planning appears to reduce cost, with small decreases in success rate in the reported setting. That is a more interesting result than a leaderboard number, because it says the same harness feature can have different value depending on the model behind it.

It also compares predefined tools with Bash-only setups. In the paper's reported results, predefined tools help models with weaker Bash control, while Bash-only can produce higher success at lower cost for models that are already good at shell-centric work.

The trap is reading that as a universal rule. The study uses Nemotron and Mistral model families, and it does not include every model a working AI coding team might care about, such as Qwen, DeepSeek, Claude, or GPT-class production systems. I would treat the pattern as portable, but not the exact numbers.

Planning is not automatically good

Planning sounds harmless. In agentic coding, it is often useful because it turns a vague task into steps the agent can inspect, revise, and finish. It can also become ceremony if the model already has enough internal task control for the job.

That is why the paper's planning result is worth paying attention to. A planning tool can lift a weaker model by making the work less implicit. With a stronger model, the extra structure may mostly change cost and behavior rather than raw success.

A concrete Codex example is a dependency update. Asking an agent to first list files, then edit, then run tests may save wandering when the codebase is unfamiliar. On a one-line API rename, the same planning step can be slower than letting the agent search, patch, and verify directly.

A small repo rule helps here. Put durable constraints in AGENTS.md, not in an every-task planning prompt. Planning should describe the task at hand, while AGENTS.md should carry the repo's standing rules about architecture, test commands, and review expectations.

Tool choice changes what success means

Predefined tools make a task easier to steer. A tool such as search symbols, update issue state, or read dependency metadata narrows what the agent can do and gives it a cleaner interface than a raw terminal.

Bash gives freedom. It also asks the model to know shell syntax, repo layout, quoting, process control, exit codes, and how not to delete the wrong thing. A model that is strong at shell use may benefit from that freedom, especially on tasks that are naturally command-heavy.

Here is a small AGENTS.md boundary I would use in a Codex repo when comparing the two styles:

# Agent instructions

- Prefer repository scripts over ad-hoc commands.
- Before editing generated files, find the generator and update that first.
- Use `npm test -- --runInBand` for the focused verification pass.
- Do not run destructive shell commands without an explicit human confirmation.
- Summarize changed files and failing checks before asking for review.

That file does not make the agent smarter. It makes the harness more legible. If a Bash-heavy run succeeds only because the agent guessed hidden conventions, you have learned less than you think.

MCP turns harness design into architecture

MCP, the Model Context Protocol, is the integration layer many teams use to connect agents to tools such as repositories, issue trackers, databases, documentation, and design systems. Once an agent can call an MCP server, harness design stops being only a benchmark concern. It becomes a production boundary.

The question is not simply whether an MCP server exists. The question is what it can read, what it can mutate, and how its output is checked. A read-only documentation server is a very different harness choice from a server that can create branches, update tickets, or write to a database.

One concrete pattern is to make the first MCP server read-only. Let Codex inspect issues, docs, or code intelligence, then require normal Git changes and review for mutations. If your agent needs richer code navigation, the same boundary question appears in projects like graphify-csharp Gives Agents C# Find Usages, where the value is not magic autonomy but better retrieval at the right moment.

The limitation is latency and trust. Every tool call adds time, context, and a new failure mode. A harness with ten tools can look powerful while making it harder to explain why the final patch happened.

Try a small harness comparison

You do not need to recreate the paper to learn from it. Pick one normal task and run it twice with the same model, same branch base, and same success criteria. Change only the harness.

Use a task that has a clear finish line, such as fixing a failing test, adding one validation rule, or replacing a deprecated API. Avoid broad refactors. The point is to observe harness behavior, not to stage a heroic demo.

Copy this light experiment plan into your issue or handoff note:

## Harness comparison

Task: <one small coding task with a clear test>
Base branch: <branch or commit>
Model: <model name, unchanged between runs>

Run A, planned:
- Ask the agent to inspect first and write a short plan.
- Allow repo search and normal shell commands.
- Require a test command before final answer.

Run B, direct:
- Ask the agent to patch directly.
- Allow the same files and same commands.
- Require the same test command before final answer.

Record:
- Did it pass the target check?
- How many files changed?
- Which commands ran?
- Did it ask for help at the right time?
- Would you merge it after review?

For Codex CLI, keep the verification loop boring on purpose:

git checkout -b harness-measurement-demo
# run the agent once with the planned prompt
npm test -- --runInBand
# reset to the same base, then run the direct prompt
git reset --hard <base-commit>
npm test -- --runInBand

Do not compare vibes. Compare the patch, the commands, the failed or passing checks, and the review burden.

Common questions

• Does this prove one coding model is better than another?

  No. The paper is mainly evidence that harness choices can materially change measured coding-agent performance. The useful caution is that planning, predefined tools, Bash access, and stopping rules can all affect success, cost, and failure shape before you ever get to a clean model comparison.

• Should Codex agents use Bash-only workflows?

  Not by default. The paper suggests Bash-only can work well for models that are already strong at shell-centric tasks, but predefined tools can help models with weaker Bash control. In a repo, use Bash where the command line is the natural interface and add tool boundaries where precision or permissions matter.

• Is planning still worth using?

  Yes, when the task has uncertainty that benefits from a visible intermediate step. The paper's reported pattern is mixed: planning helps weaker models at extra cost, while for stronger models it can reduce cost with a small success tradeoff in that setting. Treat planning as a harness option, not a moral good.

• How does this relate to MCP?

  MCP is one way to make harness choices real in production. An MCP server can give an agent safer access to docs, code search, tickets, or other systems, but it also defines what the agent can see and do. That boundary should be reviewed as carefully as a shell permission.

• What is the biggest limitation of the study?

  Model coverage is the obvious caveat. The reported experiments use Nemotron and Mistral models, so the exact outcomes should not be copied onto every current frontier or open model family. The stronger lesson is methodological: compare agents with the harness held visible, not hidden.

What to inspect before trusting an agent result

The table is not a scoring rubric. It is a way to keep the comparison honest. If two agent runs use different planning rules, different tools, or different permissions, the model is not the only thing being measured.

Further reading

• An empirical study of harness design for coding agents
• Codex Agent docs
• MCP specification
• OpenAI Codex documentation

Have you tried these

Run the change on one branch. Then use our methodology at Review, an engineer still owns the merge, or use hands-on training when you want the team version of that step.