Evaluate Prompt Quality in Claude Code and Codex

How to Quantitatively Evaluate Prompt Quality in Claude Code and Codex is a Medium post by Koukyosyumei about measuring prompts for Claude Code, Anthropic's coding agent, and OpenAI Codex rather than judging them by feel. It deals with a very practical question: did this prompt make the agent produce a better code change, or did it only sound better in chat? The useful takeaway is to turn prompt quality into a small eval loop: same repo, same task, same checks, scored output. A Codex agent is OpenAI's coding agent, and in the agents md codex workflow its durable repo context belongs in AGENTS.md, not in every prompt you are trying to grade.

What Koukyosyumei measured

The interesting move in Koukyosyumei's post is not that prompts matter. Everyone using coding agents already knows that.

The move is treating a prompt as an engineering input that can be tested. A prompt is not just a sentence you like. It is a control surface that changes the diff, the tests, the amount of unnecessary code, and the review burden.

That is why developers paid attention. Claude Code and Codex can both produce plausible work from vague instructions, but plausible is not the same as correct. Once an agent can edit a repo, run commands, and explain its choices, the quality bar moves from nice answer to safe patch.

Prompt quality is the degree to which a prompt reliably produces the intended code change with minimal extra damage. That definition matters because it pushes the measurement away from vibes and toward artifacts: patch, test log, transcript, and reviewer notes.

The trap is comparing two prompts by reading two agent replies. The reply is only the surface. The real object is the resulting change in the repository.

Turn one coding request into a repeatable test

Start with one boring repo task. Good eval tasks are small enough to run repeatedly and real enough to expose agent behavior.

For example, use a failing test in a TypeScript service:

mkdir -p .prompt-evals/runs

git switch -c eval/prompt-quality

git reset --hard main

printf '%s\n' 'Fix the failing user-settings test. Keep the public API unchanged. Run the focused test and explain any tradeoff in the final note.' > .prompt-evals/prompt-a.txt

codex 'Fix the failing user-settings test. Keep the public API unchanged. Run the focused test and explain any tradeoff in the final note.' | tee .prompt-evals/runs/a-01.log

npm test -- user-settings | tee .prompt-evals/runs/a-01-tests.log

git diff --stat > .prompt-evals/runs/a-01-diffstat.txt
git diff > .prompt-evals/runs/a-01.patch

If your Codex CLI setup uses an interactive session, paste the exact same prompt and save the transcript manually. The important part is not the shell trick. The important part is that Prompt A and Prompt B face the same repo state and leave comparable receipts.

Then reset and run the next prompt:

git reset --hard main

printf '%s\n' 'Repair the user-settings failure with the smallest safe code change. Do not rename exported functions. Run the focused test before finishing.' > .prompt-evals/prompt-b.txt

codex 'Repair the user-settings failure with the smallest safe code change. Do not rename exported functions. Run the focused test before finishing.' | tee .prompt-evals/runs/b-01.log

npm test -- user-settings | tee .prompt-evals/runs/b-01-tests.log

git diff --stat > .prompt-evals/runs/b-01-diffstat.txt
git diff > .prompt-evals/runs/b-01.patch

This is a small Codex CLI workflow, but it is enough to learn something. If Prompt B fixes the same bug with a smaller patch, a passing focused test, and fewer invented assumptions, it probably beats Prompt A for this task type.

The trap is changing two things at once. Do not change the prompt, repo state, test command, and agent permissions in the same comparison. You will not know what caused the difference.

For more CLI patterns, keep this close to the related training topic, where repo-local instructions and verification loops matter more than one perfect prompt.

Keep repository memory separate from the prompt

AGENTS.md is where Codex reads durable project guidance. Put stable rules there: architecture boundaries, test commands, formatting expectations, and safety notes.

Do not hide those rules inside the prompt you are scoring. If Prompt A includes a secret paragraph about the test command and Prompt B does not, you are not measuring prompt quality anymore. You are measuring missing context.

A tiny AGENTS.md boundary for eval work can look like this:

# Agent instructions for prompt eval runs

- Prefer the smallest code change that fixes the named failure.
- Do not change public APIs unless the prompt explicitly asks for it.
- Run `npm test -- user-settings` before claiming success.
- Treat GitHub, Jira, Slack, and production databases as read-only during eval runs.
- If an MCP server exposes write actions, ask before using them.

This is the practical agents md codex point: AGENTS.md should define the playing field, while the prompt under test should define the task. That separation makes the result easier to compare.

Nested AGENTS.md files are useful when only part of the repo has special rules. A frontend package can say which Playwright command to run. A backend package can name the database fixture policy. Local scope beats a giant root file that every task has to carry.

The trap is letting memory files become a second prompt arms race. Keep them short. If the instruction would change from task to task, it probably belongs in the evaluated prompt or the issue, not in AGENTS.md.

Score the diff, not the conversation

A good scorecard is dull on purpose. It should reward the things a reviewer already cares about.

Use five checks for a first pass:

Check	Score	What to look at
Task completion	0-3	Did the change solve the named problem?
Test evidence	0-3	Did the expected test run, and did it pass?
Patch size	0-2	Was the diff appropriately small?
Constraint respect	0-2	Did it preserve APIs, style, and repo rules?
Review clarity	0-2	Did the final note explain what changed without noise?

Keep the maximum small. A twelve-point score is easier to use than a fake-scientific hundred-point scale.

Run each prompt more than once if the result matters. Coding agents can vary across runs because context, tool calls, and model behavior are not perfectly deterministic. Three runs per prompt is often enough to reveal an obvious bad prompt; it is not enough to publish a universal benchmark.

This is also where sandboxing matters. If your eval lets an agent write to external systems through MCP, the measurement can cause real damage. For heavier experiments, the same idea behind Sanbox Gives AI Agents MicroVM Sandboxes applies: isolate the agent before you trust the result.

The trap is scoring the prettiest explanation. A fluent final answer with a messy diff is still a bad coding run.

Try it safely with a fit/not-fit table

Use this table when deciding whether a prompt-quality eval is worth running. Copy it into an issue before spending an afternoon on prompt variants.

Situation	Fit?	Why
One repeated bug-fix or refactor task keeps appearing	Fit	You can compare prompts against the same kind of work.
The repo already has a focused test command	Fit	Test logs give the scorecard a hard signal.
AGENTS.md contains stable repo rules	Fit	The prompt can stay focused on the task instead of restating project law.
You are choosing between two Codex prompts for the same workflow	Fit	Same agent, same task, different prompt is a clean comparison.
You are comparing Claude Code and Codex as products	Partial fit	Keep the task and scorecard fixed, but expect tool differences to matter.
The task needs judgment from product, design, or security	Partial fit	Add human review notes; tests alone will under-score risk.
The repo has no tests and no review criteria	Not fit yet	Write the check first, then measure the prompt.
The prompt requires broad exploration across many files	Not fit for a first eval	The result will be noisy and hard to attribute.
MCP tools can mutate external systems during the run	Not fit until bounded	Make those connections read-only or run in a safe environment.

The smallest useful experiment is two prompts, one task, three runs each, and one scorecard. That is enough to find bad wording, hidden assumptions, and instructions that make the agent over-edit.

The trap is turning the first eval into a benchmark platform. Do the tiny version first. You can always add more tasks after the scorecard catches one real difference.

Common questions

• What is Codex agent in this workflow?

  A Codex agent is the OpenAI coding agent doing the repo work: reading files, proposing edits, and running commands through the Codex experience you use. In this workflow, the agent is the executor, while the prompt, AGENTS.md file, test command, and scorecard are the controlled inputs around it.

• How should I use agents md codex when measuring prompts?

  Use AGENTS.md for durable repository rules and keep the evaluated prompt focused on the task. That means test commands, API constraints, MCP boundaries, and style rules can live in AGENTS.md, while Prompt A and Prompt B differ only in the instruction style you want to compare.

• Should I compare Claude Code and Codex with the same prompt?

  Yes, but treat it as a tool comparison, not a pure prompt comparison. The same prompt can be useful across Claude Code and Codex, but different context handling, tool behavior, and defaults can affect the result. Keep the repo state, task, tests, and scoring rubric identical.

• Where do I find the openai codex-1 agent maximum context tokens 2025 number?

  Check current OpenAI documentation rather than baking a cached 2025 number into your eval. Context limits and product surfaces can change, and prompt-quality tests should record the date, model or agent name, and relevant settings used for each run instead of assuming an old limit.

• Do I need a big benchmark to learn anything?

  No. A small benchmark with one real task can reveal whether a prompt causes over-editing, skips tests, or ignores constraints. The caveat is scope: a prompt that wins on a bug fix may not win on a migration, design change, or security-sensitive review.

Best ways to use this research

• Best for: debugging prompts that already run against a real repo, especially repeated Codex workflows like bug fixes, test repairs, and small refactors.
• Best first artifact: a .prompt-evals/ folder with the prompt text, transcript, patch, test log, and a twelve-point scorecard for each run.
• Best comparison angle: compare two prompts inside one tool first; compare Claude Code and Codex only after the task, repo state, and scoring method are stable.
• Best safety boundary: keep MCP connections read-only during eval runs unless the task explicitly needs writes and you have a disposable environment.

Further reading

• OpenAI Developers — AGENTS.md guide
• OpenAI Developers — Codex quickstart
• How to Quantitatively Evaluate Prompt Quality in Claude Code and Codex — source

Next step

Pick one failing test and run two prompt variants against the same clean repo state. Keep the better prompt only if the diff, test log, and review score agree.

One methodology lens

One useful way to read this through our methodology is the Plan step: delegate first-pass decomposition and dependency mapping, review the sequencing and assumptions, and keep ownership of scope and priorities. If that split is still fuzzy, the workflow usually is too.