AI Coding Agents Can Increase Output. But Are We Measuring the Right Outcome?
Software teams have spent years trying to reduce the time between an idea and a working release. AI coding agents appear to compress that timeline dramatically.
They can inspect a repository, draft an implementation, write tests, update documentation, open a pull request, and respond to review comments. The visible output increases quickly: more commits, more pull requests, more changed files, and more tasks moved across the board.
That sounds like productivity. Sometimes it is.
But after years of building software systems, I have learned to be careful with metrics that reward motion without proving value. A team can produce more code and still ship the wrong feature. It can merge more pull requests and quietly increase operational risk. It can shorten implementation time while creating a larger review, security, and maintenance burden downstream.
🟦 Operator View: The purpose of an AI coding agent is not to maximize code production. It is to help a team deliver reliable customer value with less avoidable effort.
What the latest research actually tells us
A July 2026 study of tens of thousands of engineers during Microsoft’s early rollout of Claude Code and GitHub Copilot CLI found that adopters merged roughly 24% more pull requests than they otherwise would have. The researchers explicitly acknowledged an important limitation: a merged pull request is a proxy for output, not a direct measure of the value delivered by that change.
That distinction is the center of the conversation.
The study gives us meaningful evidence that command-line coding agents can change engineering throughput at organizational scale. It also shows that adoption and retention are uneven. Engineers were more likely to continue using the tools when the tools fit their real coding activity, and visible use by peers helped adoption spread.
Other 2026 research has started examining how agent-generated pull requests differ from human contributions, how reviewers respond to them, and why some agent-authored fixes remain unmerged. Common failure patterns include test failures, duplicate work, repository-context mistakes, and changes that look plausible but do not fit the maintainer’s actual intent.
The lesson is not that AI coding agents do not work. The lesson is that output is only the beginning of the measurement system.
Read the Microsoft rollout study and compare it with emerging research on how coding agents modify code and why agent-involved fixes remain unmerged.
The metric trap: counting what the agent makes
Engineering organizations naturally count what their tools make visible. That usually means:
- Pull requests opened or merged
- Commits generated
- Lines of code added
- Issues closed
- Story points completed
- Time from task assignment to first implementation
These metrics are useful operational signals. They can show whether an agent is being used and whether work is moving. The problem begins when they become the definition of success.
More code is not automatically better software. A smaller change that removes complexity can be more valuable than a large generated implementation. A pull request that merges in ten minutes can still cause a support problem next week. An agent may complete a ticket exactly as written while missing the product context that would have revealed the ticket was unnecessary.
🟥 Measurement Warning: When the metric rewards production volume, the system will produce volume. It will not automatically produce clarity, reliability, or customer value.
A better way to measure AI coding agents
I would evaluate coding agents across five layers. The first two tell you whether the agent is active. The final three tell you whether the agent is useful.
| Measurement layer | What to track | What it tells you |
|---|---|---|
| Adoption | Weekly active users, repeat usage, task types delegated | Whether engineers find the tool relevant enough to reuse |
| Throughput | Cycle time, PR volume, lead time to first review | Whether work moves faster through the development process |
| Quality | Escaped defects, rollback rate, test reliability, security findings | Whether speed creates hidden rework or production risk |
| Human load | Review time, correction rate, cognitive load, interruption frequency | Whether the agent removes work or merely relocates it |
| Business outcome | Time to customer value, adoption of shipped features, support impact, revenue or cost outcome | Whether the software change mattered outside engineering |
1. Measure acceptance, not just generation
An agent can generate ten proposed changes. That does not mean ten changes were helpful.
Track how often generated work survives human review without major rewriting. Separate minor formatting corrections from architectural corrections. A high generation rate with a high rewrite rate may look productive in the agent dashboard while consuming senior engineering time.
A useful metric is accepted work per review hour. It connects the agent’s output with the human effort required to make that output trustworthy.
2. Measure rework after merge
The first merge is not the end of the task. Track what happens in the next seven, fourteen, and thirty days.
- Was the change reverted?
- Did it produce a related incident?
- Did another engineer need to rewrite it?
- Did support tickets increase?
- Did the change add maintenance complexity?
If an agent increases pull-request volume by 24% but also increases corrective work, the net productivity gain may be far smaller than the headline suggests.
3. Measure the review bottleneck
AI coding agents make code generation cheaper. That shifts the bottleneck.
GitHub has described this directly: generating code is becoming easier, while reviewing, securing, governing, and deploying software remain the harder parts of the lifecycle. When agents can create work faster than people can validate it, the organization does not eliminate the bottleneck. It moves the bottleneck into the review queue.
This is why engineering leaders should monitor:
- Median time to first meaningful review
- Review comments per agent-generated change
- Senior engineer time spent validating generated work
- Security and compliance review latency
- Queue size for agent-created pull requests
An agent program is not healthy when developers generate more work while reviewers become the exhausted control plane.
4. Measure product impact
The most important question is simple: did the change improve the product or the business?
Connect engineering work to outcomes such as:
- Reduced time for a user to complete a task
- Higher activation or retention
- Lower support volume
- Faster onboarding
- Improved reliability
- Lower infrastructure or operational cost
- Revenue protected or enabled
This is harder than counting pull requests, but it is the level at which leadership decisions should be made.
The same principle appears throughout AI operations: access to a tool is not the advantage. Operational execution is. I explored that wider gap in The Playbook for a $100M AI Agency and in my notes on measuring AI automation through real customer-service outcomes.
The developer’s job is changing, not disappearing
Coding agents do not remove the need for experienced engineers. They increase the value of judgment.
When implementation becomes cheaper, the scarce skills become:
- Defining the right problem
- Breaking work into safe boundaries
- Designing architecture and interfaces
- Writing acceptance criteria
- Reviewing security and failure modes
- Understanding customer and business context
- Deciding what should not be built
Junior engineers may generate working code earlier. Senior engineers may supervise more parallel work. But neither group becomes optional. The team needs people who can recognize when an apparently correct implementation is wrong for the system.
đźź© Practical Principle: AI lowers the cost of producing an answer. Engineering still owns the cost of being wrong.
How I would introduce coding agents into a real team
Start with bounded, verifiable work
Do not begin by giving an agent ownership of a critical system. Start with work that has clear inputs, clear tests, and a low blast radius.
Good early tasks include:
- Test generation for existing behavior
- Documentation updates
- Small refactors with strong test coverage
- Internal scripts and migration helpers
- Dependency updates in isolated services
- Bug reproduction and diagnostic summaries
Create an agent-specific review policy
Agent-generated code should not be treated as suspicious by default, but it should not be treated as trusted by default either.
Define which changes require human approval, security review, performance testing, or staging validation. Record the tool and model used. Keep prompts, task context, and execution logs for important changes. Make rollback paths part of the workflow.
Compare teams using outcome bundles
Avoid comparing one team’s PR count with another team’s PR count. Compare bundles of outcomes over a meaningful period:
- Lead time improved
- Defect rate stayed stable or fell
- Review burden did not increase disproportionately
- Customer or operational metrics improved
- Engineers report lower repetitive workload
That gives you a more honest picture of productivity than any single dashboard number.
The question leaders should ask
“How much more code did we produce?” is an easy question.
A better one is:
Did AI help us deliver the right change, with less total effort, without transferring unacceptable risk to reviewers, customers, or operations?
That question forces engineering, product, security, and business leaders to share one measurement system.
AI coding agents are real leverage. The Microsoft study is useful evidence that they can increase engineering output at scale. But the mature response is not to celebrate every increase in generated activity. It is to build the evaluation layer that distinguishes useful acceleration from faster noise.
Frequently asked questions
Do AI coding agents actually improve developer productivity?
Recent large-scale research suggests they can increase measurable engineering output, including merged pull requests. The size of the true productivity gain depends on quality, review burden, rework, and whether the changes create business value.
Is pull-request volume a good productivity metric?
It is a useful throughput metric, but not a complete productivity metric. Teams should combine it with defect rates, review time, rollback rates, customer impact, and total effort.
What should engineering leaders measure first?
Start with adoption, accepted work per review hour, cycle time, post-merge rework, escaped defects, and the product or operational outcome connected to each class of work.
Will AI coding agents replace software engineers?
They are more likely to change the engineer’s role by shifting effort from manual implementation toward problem definition, architecture, review, security, and system-level judgment.
Final takeaway
AI coding agents should be evaluated as part of a delivery system, not as isolated code generators.
Measure what they produce. Then measure what survives. Finally, measure what matters.
For more grounded notes on AI systems, SaaS building, and production engineering, explore the latest writing at turjo.me.
