Setting a codex SLI your team will actually watch

Why a codex SLI is worth defining

Service level indicators come from operations: pick a measurable signal, watch it, argue about it with data instead of anecdote. Apply the same idea to agent-assisted work. A codex SLI is a number you agree to track about the output of the agent, not about how much people like it.

Most teams we work with have no measurement at all six weeks in. They have a strong opinion instead, usually formed by one bad afternoon. That opinion then decides a six figure tooling budget.

Four indicators worth the effort

• First-diff acceptance rate. Of agent-produced changes, what fraction got merged without the author rewriting the approach. Not without any edits. Without a rewrite.
• Review time per merged line. If agent pull requests take three times longer per line to review, the speed gain is fake and has moved onto someone else's calendar.
• Rework within fourteen days. Count changes to files an agent touched, made by a human fixing behaviour. This is where confidently wrong work shows up.
• Test suite pass on first push. Cheap to collect from CI, and it moves quickly when your instruction file improves.

Four is enough. Teams that pick twelve indicators check none of them by month two.

How to collect it without building a platform

Do not start with a dashboard. Start with a label. Tag agent-assisted pull requests with something like agent-assisted and query them later. A single command gets you a baseline:

gh pr list --label agent-assisted --state merged --limit 100 --json number,createdAt,mergedAt,additions,deletions

That gives you cycle time and size. Pair it with a two-question exit poll on merge, answered in the PR description, and you have acceptance and rework without any infrastructure. We have run this by hand for a quarter at a fintech before anyone wrote a script.

The indicators that mislead

Lines of code generated is the worst one. It rewards verbosity, and agents are already verbose. Number of tasks run is close behind. Both go up when the tool is being used badly.

Suggestion acceptance from an editor completion counts a keystroke, not a decision, so it drifts far from anything a director cares about. Time saved, self-reported, is a survey of optimism. We have seen the same team report 40 percent time savings while their median cycle time was flat.

Be honest about a limit here too. None of these indicators separate a good change from a change that merely passed. A codex SLI tells you where to look. It does not tell you the code is right.

What to do next

Pick two indicators, not four, for the first month. First-diff acceptance and rework are the pair we recommend, because together they catch the failure everyone fears: fast output that costs more later. Set the label today, collect for four weeks with no target attached, then look at the numbers as a team before anyone sets a goal.

Targets set before you have a baseline get gamed. Every time.

If you want help putting this into practice, talk to us.