The Half We Don't Measure

Ask most companies how AI is going and you'll get a usage number back. Seats filled, tokens burned, percentage of engineers active in the tool, PRs opened. Those are all real numbers. What they don't tell you is whether any customer ended up better off.

That's the gap I want to talk about. If you're going to incentivize AI use, the thing worth incentivizing is a better outcome for the customer. Usually that shows up as speed. It can also show up as a lower defect rate, a system that holds up under load, or someone catching a problem in review that would otherwise have shipped. Everything else we're measuring is a proxy, and we mostly picked proxies that sit at the wrong end of the pipeline.

Throughput was a good signal once

Attach a number to someone's performance review and people start managing the number instead of the thing it was supposed to describe. Usage metrics were genuinely useful right up until they landed on a dashboard. Back when using AI was voluntary, someone with high usage was just someone excited about the tool. Once a manager could see the number, it started measuring something closer to willingness to look like you're on board.

PR throughput is the more interesting case, because there's real research behind it. It's been part of validated delivery work for over a decade and it does correlate with organizational outcomes. [1]

The catch is that research was done when writing code was the bottleneck, and the bottleneck has moved. Pair a throughput target with an AI usage push and you've told people to generate more code, which they'll happily do. Whether that gets a customer anything is a separate question, and sometimes the answer is that it costs you, because every line you merge is a line somebody now owns.

Where the incidents come from

This part tends to get left out of AI adoption conversations.

Two years of engineering telemetry covering 22,000 developers across more than 4,000 teams found real output gains under heavy AI adoption. Task completion up 34%, epics completed per developer up 66%. Those came out of telemetry rather than a survey, which makes them about the strongest evidence I've seen that this stuff works.

The same dataset has another side to it. Incidents per pull request rose 242.7%. Bugs per developer rose 54%. Median time in PR review rose 441%, and 31% more PRs merged with no review at all. [2]

That last number explains a lot of the others. Code generation got faster than review, testing, and deployment could absorb, so the reviewing stopped happening. AI-generated code is also unusually good at surviving the review that does happen. It reads cleanly, it's well formatted, and the comments are plausible. It passes the sniff test in a PR and then fails against the parts of your system the model never saw. That makes it easy to trust enough to ship without reading line by line, and you find out in production.

The cost of that lands on whoever's on call, who usually isn't the person who merged it. Which makes it an incentive problem. If the person generating the code gets credit for throughput while someone else absorbs the incidents, you've pulled the reward apart from the consequence, and you should expect exactly what you get.

Somebody has to maintain all of this

Incidents show up within days. The maintenance cost takes years to fully arrive, which is part of why it's so easy to ignore.

Research covering hundreds of millions of real code changes shows duplicated code blocks climbing every year since 2023, now at the highest level on record. Over the same period refactoring collapsed. The share of moved, or refactored, lines fell from 21% in 2022 to 3.8% so far in 2026, while copy and paste rose from 9.4% to 15.7%. Developers are now roughly five times more likely to duplicate than refactor, an inversion of the two-to-one preference for refactoring that held before these tools showed up. [3] AI adoption isn't the only thing that changed in that window, so I'd hold the causation loosely.

Every copy of a block is one more place a future change has to reach, and tracking them all down falls to whoever touches that area next, usually someone who wasn't there when it was written. Multiply that across a codebase that's growing faster than anyone can read it.

It shows up in capacity. An analysis of over a million PRs across thousands of engineering organizations put reactive work, meaning bugs plus maintenance, at 44% of capacity for the median org, with reverts growing faster than PR volume. [4]

You can argue with any specific number in here, and you should discount the ones coming from vendors who sell into this problem. The direction is harder to wave off, since it shows up across datasets that don't share a methodology. We built services faster than we built the capacity to run them.

So count the whole cost before you call something a win. A feature shipped in half the time that triples on-call load and permanently raises the cost of every future change in that area did not make anyone faster. It moved the work somewhere nobody's measuring.

The test I'd actually use

This is simple enough to put in front of a leadership team. As the volume of merged code goes up, these should hold flat or improve:

CSAT, or whatever your honest read on customer sentiment is
Incident rate, and change failure rate alongside it
Core observability metrics: latency, error rate, saturation, the things you already have dashboards for

If code volume is climbing and any of those are degrading, the speed is coming out of somewhere, and it's usually reliability.

You already have every input for this. There's no new AI dashboard to buy. The hard part is putting the throughput chart next to the incident chart and looking at them together, which almost nobody does, because they live in different tools and get reported by different people to different audiences.

The test also doesn't care how much AI was involved. If a team hits its outcomes with no AI at all, it says they succeeded, because they did. That's deliberate. The moment your measurement system starts caring about AI usage for its own sake, you're back where you started.

Reviews drift toward whatever is easiest to see

The same problem shows up in performance management, and it's the expensive version.

A demo instruments itself. It has a moment where everyone says "oh, nice," it fits in a Slack post, and it travels without anyone doing extra work. Someone who wires up an agent over the weekend and posts a two minute video has produced a clean, legible artifact of contribution.

The work that moves your business usually produces nothing. A quarter spent making onboarding not suck for a segment that was quietly churning shows up as an absence. The incident that didn't happen, the ticket nobody filed, the customer who renewed without a conversation. If nobody instrumented retention for that segment, the person who fixed it walks into a review with no artifact at all.

So the review cycle drifts toward whatever is easiest to see, and a usage metric accelerates the drift. You end up promoting against a proxy with no established relationship to business results, in parallel with the work that actually produces them. Both groups notice which one the company pays for, and people are rational about that.

This one is fixable, and it's your measurement problem rather than their contribution problem. If you can't name the customer outcome someone moved this cycle, that's usually a gap in what you instrumented. Point the same metrics from the last section at people: which team moved time to value, which one drove down escaped defects, which one absorbed the on-call load somebody else generated. Calibrate against that instead of against activity. It's the same data either way, and you're already collecting it.

What I'd do instead

Tie the incentive to customer outcomes. Time to value for a new customer, resolution rate, escaped defect rate, cycle time from request to something the customer can actually use. Speed and quality both count here, and a team that used AI to ship the same volume with half the defects delivered real value. If AI helps, these move. If it doesn't, you've learned something more useful than a usage report.
Pair every throughput metric with a stability metric. Never show PR volume without change failure rate and incident rate next to it. Untethered throughput targets are how you get the incident spike.
Make the generator own the consequence. If AI-assisted code raises on-call load, that has to land on the team that shipped it rather than diffusing into a platform group. Google's SRE practice has enforced this for years: operational work is capped at 50% of an SRE team's time, and the overflow gets redirected back to the product development team by reassigning tickets and putting developers back into the pager rotation until the load comes down. They describe it as "an effective feedback mechanism, guiding developers to build systems that don't need manual intervention." [5] AI has made that loop matter a great deal more than it used to.
Budget for maintenance explicitly. Refactoring is down 5x while duplication is at record highs. That debt compounds. If your planning assumes AI-generated code is free once it's merged, you'll keep running short on capacity and not know why.
Reward the practice itself. A study of agent trajectories across 103 engineering teams found that cost, autonomy, and rework had far less to do with which model was used than with three things the team controls: how clearly the task was specified, whether the repo and tests let the agent verify its own work, and whether the developer steered the output toward production standards. A one-point gain on that last one was associated with 39% fewer review cycles. [6] Those are teachable habits, and they're a much better thing to reward than usage.
Separate "are people using this" from "is this working." Track adoption if you want, as a signal about enablement and training. Keep it out of performance reviews.

The tool was never the problem

I'm not skeptical about the technology. I use these tools every day and they've changed how I work.

The problem is that we started running organizations against a proxy. Usage, tokens, PR count: none of these have an established relationship to whether the business is winning, and every one of them can be moved a long way without a customer noticing anything at all.

AI mostly amplifies what a company already is. If you were clear about customer outcomes before all this, you'll get there faster now. If you were confused about what you were building, you're now confused at a much higher token spend, with more incidents and more code to maintain.

So measure the outcomes directly, which you already have the instrumentation to do. Then fund the people and the processes that move them, whether or not the work looks like AI adoption from the outside. The proxies will keep looking good either way, and that's the whole problem with them.

References

DORA, which has studied software delivery performance since 2014 and treats deployment frequency, lead time, and rework rate as throughput measures validated against organizational outcomes.
Faros AI, The Acceleration Whiplash: AI Engineering Report 2026, based on two years of telemetry across 22,000 developers and 4,000+ teams. Faros sells engineering analytics, so treat the framing accordingly, though these are measured signals rather than survey responses. Directionally consistent with DORA's 2025 report, which found AI adoption correlating with higher throughput and higher instability at the same time.
GitClear, The Maintainability Gap: 2026 AI Code Quality Research, based on 623 million changed lines. The 2026 figures are year to date and the 2022 baselines come from their earlier reports. GitClear sells code quality analytics, and their "moved line" classification is proprietary rather than peer reviewed.
Entelligence, Token Maxxing Is Making Engineering Teams Slower, based on over 1 million PRs across 2,444 organizations. Entelligence sells AI code review tooling and the methodology isn't published in detail, so treat this as directional.
Beyer, Jones, Petoff, and Murphy, Site Reliability Engineering, Chapter 1, on ensuring a durable focus on engineering. The book describes the redirection as a safety valve, and notes it works only when the whole organization understands why it exists.
Span, Beyond the Model: What Distinguishes Effective Agentic Development, covering complete agent trajectories from 103 engineering teams over 12 weeks in mid-2026. They also found a one-point gain in prompt clarity associated with roughly 27% lower token cost per merged AI-authored line, and a one-point gain in environment readiness with roughly 88% more merged AI code per human turn. Span notes these are observational rather than causal, and Span sells engineering analytics.
METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. In February 2026 METR published new data and changed its experiment design, stating the original results no longer reflect current AI impact. Their newer estimates point toward a modest speedup with wide confidence intervals, and they consider the signal unreliable, partly because developers increasingly decline to participate in a study that requires working without AI.