AI Developer Productivity Is Not a Headcount Argument

Nearly every conversation I have about AI developer productivity arrives at the same place inside ten minutes. Someone asks how many engineers it saves. Not what it makes possible. How many people it removes. Efficiency has been quietly narrowed until it means one thing, and that one thing has the thinnest evidence behind it of anything in the current debate.

I have spent eight years inside China's technology sector, most of them writing and shipping software before moving into AI and technical leadership. Shenzhen is a useful place to watch this from. Hardware culture here has a long memory for the difference between a cost reduction and a capability, because the two look identical on a quarterly sheet and behave nothing alike over five years. One is a subtraction you can perform once. The other changes what the next product can be.

My argument is narrow. AI-assisted engineering produces real gains. They are mostly not the gains being sold. They show up as a raised floor rather than a shrunken team: fewer defects reaching production, faster orientation in an unfamiliar codebase, work that finally gets done because somebody had the hours. Those gains compound. Headcount savings do not.

Four studies, four different answers

Start with the result that embarrassed everyone who wanted a clean story. METR ran a randomised controlled trial in early 2025 with 16 experienced open-source developers across 246 real issues in repositories they already maintained. Developers using AI tools took 19 percent longer. Their own forecast beforehand was a 24 percent speedup. Afterwards, having measurably been slower, they still believed they had been about 20 percent faster.

METR are careful about what that does and does not show. They state plainly it is not evidence that AI fails to speed up most developers, and they flag likely bias in who volunteered. Fair. Now put it beside Google's own enterprise trial, published in late 2024, where 96 full-time engineers worked a complex internal task. Best estimate: roughly 21 percent less time on task, with a confidence interval the authors themselves call large and an explicit warning that internal Google tooling in the summer of 2024 may not generalise across tools or over time.

The largest headline number comes from the most artificial setting. Peng, Kalliamvakou, Cihon and Demirer found a 55.8 percent speedup for developers given GitHub Copilot. The task was implementing an HTTP server in JavaScript as fast as possible. Greenfield, self-contained, no legacy, no reviewers, no consequences. Then Uplevel, a developer-analytics vendor reporting on its own customer data rather than a controlled trial, looked at close to 800 developers under production conditions and found little to no gain in cycle time or pull request throughput, alongside 41 percent more bugs introduced by the group with Copilot access.

Four studies. A 19 percent slowdown, a 21 percent speedup with wide error bars, a 56 percent speedup on a toy problem and no throughput gain at all with a defect increase attached. Anyone quoting one of these as settled is selling something. What separates them is not the tool. It is the distance between the task and real production work, and the experience of the person doing it.

One note on evidence hygiene before I go further. The most-quoted number in this entire argument is the claim that 95 percent of enterprise generative AI pilots produce no measurable return, usually attributed to MIT. I went looking for the primary document. The MIT-hosted link returns 403 and what circulates is a mirrored PDF from a research initiative rather than a peer-reviewed publication, resting on 52 organisational interviews and 153 survey responses. I am not going to repeat it, and note that it would support the case I am making. A figure resting on that much and quoted that widely is not evidence just because you like where it points. I am not going to repeat it, and note that it would support the case I am making. A figure you cannot check is not evidence just because you like where it points.

Almost right is the expensive failure mode. Code that is obviously wrong costs nothing. Code that is plausible, idiomatic and subtly incorrect costs a senior engineer's full attention.

The AI developer productivity gains that survive contact with reality

One pattern holds across the credible studies. Brynjolfsson, Li and Raymond looked at 5,172 customer support agents given a generative assistant and found productivity up 15 percent on average, measured in issues resolved per hour. The average hides the finding. Less experienced agents improved on both speed and quality. The most experienced agents saw small speed gains and small declines in quality. The tool did not lift everyone. It compressed the distribution by pulling the bottom up.

That is the shape I recognise in engineering work, and it points at three places where the gain is concrete and countable.

Defect escape rate first, because you can measure it today without buying anything. Count defects caught in review against defects found in production, per team, per quarter, before and after. If assisted review and assisted test generation are doing anything, the ratio moves and the production number falls. Boring, auditable and the only quality claim in this space I would defend in a room full of finance people.

Onboarding second. Time from a new engineer's start date to their first independently merged non-trivial change. In a large unfamiliar codebase most of that time is not spent writing. It is spent finding out where things live and why. A model that has read the whole repository answers that question well, repeatedly, at two in the morning, without costing a senior engineer forty minutes of interruption each time.

Third, and least discussed: work that previously did not happen at all. Every team carries a list of things everyone agrees are worth doing that nobody has ever had the hours for. Migration off the deprecated library. Tests for the module written in a hurry in 2021. Documentation for the service whose only expert left in March. That backlog is not tracked as a productivity metric, so it never enters the productivity argument. It is where a lot of the actual value sits.

Cost saving does not compound. Capability does.

Here is the distinction quarterly reporting is structurally bad at seeing. Cutting ten roles is a one-time subtraction. You bank it once. Next year the base is lower and the same cut is not available again. Raising the quality floor is a rate change. A team shipping fewer defects spends less time on incidents, which returns hours, which get spent on the backlog nobody had time for, which shrinks the defect surface again. Second-order effects, compounding quietly, invisible to any dashboard built to count people.

DORA's 2025 report on AI-assisted software development found 90 percent of respondents using AI at work, more than 80 percent believing it raised their productivity and 30 percent reporting little or no trust in the code it generates. It also found a positive relationship between AI adoption and delivery throughput alongside a negative relationship with delivery stability. Their framing is the most useful sentence published on this subject all year: AI amplifies what an organisation already has. Strong teams get stronger. Struggling teams get their existing problems louder and faster.

An amplifier is not a substitute. You cannot amplify a signal you have deleted.

The review burden is a transfer, not a saving

Now the part that rarely makes the business case. Agentic tools do not remove work. They move it, and they move it upward.

Stack Overflow's 2025 survey recorded 84 percent of respondents using or planning to use AI tools, across 33,662 responses. Trust did not follow adoption. Roughly 46 percent expressed distrust in the accuracy of AI output against about 33 percent expressing trust. The single largest frustration, cited by 66 percent, was AI solutions that are almost right but not quite. Second, at 45 percent, was that debugging AI-generated code is more time-consuming.

Almost right is the expensive failure mode. Code that is obviously wrong costs nothing, you delete it in four seconds. Code that is plausible, idiomatic, well-formatted and subtly incorrect costs a senior engineer's full attention, and it now arrives in far greater volume than before. Generation got fast. Verification did not, and verification is the part only your most experienced people can do.

GitClear's analysis of 211 million changed lines from 2020 to 2024, drawn from repositories at Google, Microsoft, Meta and other enterprises, found duplicated code blocks rising from 8.3 percent in 2021 to 12.3 percent in 2024 while refactored lines fell from around 25 percent of changed lines to under 10 percent. Correlation, and they do not claim otherwise. But the direction is exactly what you would predict when generation becomes cheap and review capacity stays fixed. Copying beats consolidating when the copy is free and the consolidation still needs a human to think.

Where headcount reduction is the honest answer

Everything above can be used as an excuse, so let me concede the strongest version of the other side.

In genuinely commoditised work, headcount reduction is the real gain, and dressing it up as capability is a comfortable evasion. Work where output is standardised, the quality bar is a checklist, volume is predictable and institutional memory is worth close to nothing. Tier-one ticket triage. Boilerplate integration against a stable, well-documented API. Routine transformation of a fixed document format. The Brynjolfsson result cuts both ways here. If a tool lifts a novice close to the performance of an experienced worker, then in a role where experience was the only differentiator, the economic value of that experience has genuinely fallen. That is a substitution, and calling it anything else is dishonest.

So the failure is not managers who cut. It is managers who apply commoditised-work logic to work that is not commoditised, then quote productivity percentages from studies of five-thousand-agent support floors and greenfield JavaScript exercises as if they transferred. Almost nobody says out loud that their engineering organisation is commodity work. Plenty of people budget as though it were.

The people who knew why

Which brings me to the trade I think is worst.

Codebases carry two kinds of knowledge. What the system does, which is written in the code and which a model can read completely. And why it was built that way, which is written down nowhere. The retry logic is aggressive because of an outage in 2022. The schema keeps a redundant column because a regulator asked once and may ask again. The service is split along an ugly seam because of a team boundary that no longer exists but whose data contract does. None of that is in the repository. It is in about four people's heads.

A model reading the codebase reconstructs the what with real fidelity. It has no access to the why, and it will confidently propose a clean refactor that deletes the ugly thing which was load-bearing for a reason nobody recorded. The only defence is a reviewer who was there. Cut headcount on projected AI gains and you do not get to choose which knowledge walks out. Voluntary departures skew toward people with options, which means senior people, which means the why.

Then the amplifier finding arrives on schedule. Higher output volume, less review capacity, more code entering a system fewer people understand. Fifteen months later the productivity spreadsheet still shows the saving, because nothing in it was ever designed to measure the cost.

The number nobody is measuring

Something in the METR result keeps bothering me and it is not the 19 percent. It is that the developers were slower and still believed they had been faster, by 20 percent, after the fact. Experienced engineers, doing their own work, in repositories they maintained. Their instrument was broken and they could not feel it.

Now consider who decides headcount on the basis of projected AI gains. Not the people doing the work. People one or two levels further from it, reading dashboards, operating in an environment where announcing an AI efficiency programme is rewarded by markets on the announcement rather than on the result three years later. If the practitioners' own self-assessment was off by 39 percentage points in the wrong direction, there is no reason to think an executive estimate is better calibrated. It is only less checkable.

Somebody should run the trial that would settle it. Two comparable organisations, same tools, one cuts on projected gains and one holds headcount and spends the recovered hours on the work nobody had time for, measured at three years on defect escape rate and on what shipped. We will not get that study. No board will fund the arm that does not cut. So the decision keeps being made on the strength of a feeling that has already been measured, once, under controlled conditions, and found to be wrong by about forty points.

Sources

METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

Paradis et al., How much does AI impact development speed? An enterprise-based randomized controlled trial (Google): https://arxiv.org/abs/2410.12944

Peng, Kalliamvakou, Cihon and Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot: https://arxiv.org/abs/2302.06590

Brynjolfsson, Li and Raymond, Generative AI at Work: https://arxiv.org/abs/2304.11771

Google Cloud, Announcing the 2025 DORA State of AI-assisted Software Development Report: https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report

Stack Overflow 2025 Developer Survey, AI section: https://survey.stackoverflow.co/2025/ai

GitClear, AI Copilot Code Quality: 2025 Research: https://www.gitclear.com/ai_assistant_code_quality_2025_research

Uplevel, Can GenAI Actually Improve Developer Productivity?: https://www.uplevelteam.com/blog/ai-for-developer-productivity