Software estimation with AI: the typing was never the estimate
7m read time

Software estimation with AI: the typing was never the estimate

Software estimation with AI keeps missing. One METR survey asked 349 technical workers how much faster they had become and how much more value they produced, and got 3x and 1.4 to 2x. The gap between those two answers is the estimate you are actually making.

Ask a developer how much faster AI made them and you get a multiple. Ask the same developer how much more value they produce and the number falls, usually by about half.

That gap comes from one survey.

METR ran it between February and April 2026 across 349 technical workers: 87 software engineers, 71 researchers, 129 academics and PhD students, 48 founders and managers. The result is a median self-reported speed change of 3x, and a median self-reported change in the value of the work of 1.4 to 2x.

Same people. Same tools. Same week. One question about speed, one about value, and a factor of two sitting between the answers.

If you estimate software for a living, that gap is the whole problem.

The number everyone quotes is the loosest one

Three days earlier, the same organisation published the piece that explains the gap. Task substitution and uplift, by Tom Cunningham and Parker Whitfill, argues that there is no single quantity called "how much AI sped me up". There are three, and they line up in an order:

uplift on old tasks ≤ uplift in value ≤ uplift on new tasks

Old tasks are the work you would have done in 2021. New tasks are the work you choose now, knowing an agent is available. Value is what you actually got.

Their worked example is deliberately small. An engineer splits an eight-hour day between writing documents and writing pull requests, an hour each. AI halves the cost of a PR and does nothing for documents. Measured on the old task list, that is a 33% uplift. Measured on the new one, where the engineer now writes eight PRs instead of four, it is 50%. Value lands somewhere between the two.

Widen the gap between the two activities and the bounds fly apart. In their more extreme case, where a PR goes from five hours to one, the three measures span +67%, +124% and +200%.

Three numbers, one engineer, one afternoon. All of them honest.

Here is what that does to an estimate. When somebody tells you their team is 3x faster, they are almost certainly reporting uplift on new tasks, because that is the number a human can feel. It is also the loosest of the three, which is the authors' whole point.

They hedge it properly, and so should you: the ordering holds under simplifying assumptions, and their own footnote allows that it can reverse in practice. What it reliably tells you is which of the three questions you are being handed an answer to. The survey respondents asked about value instead of speed landed at 1.4 to 2x, and they were the same people.

Every respondent answered honestly. They were just answering a different question from the one you need answered when you commit to a date.

You are not doing the old work faster

This is the part that breaks estimation specifically, and it is worse than a measurement error.

An estimate is a comparison. You are saying this new thing resembles that old thing, and that old thing took a fortnight. The whole method depends on the two being commensurable.

Task substitution removes that. Whatever multiple you apply, you are applying it to a task list you would never have written down in 2024, because half of it was uneconomic.

The dashboard nobody asked for. The refactor across forty files. The second implementation you built to compare against the first. The prototype you threw away on Thursday, which cost four hours and would have cost four days.

Some of that is real value and some of it is motion. The point is that your denominator is gone. The old ticket and the new ticket share a title and nothing else.

I have written before that the aggregate speed number stopped meaning anything once developers started choosing, per task, where they wanted the help. This is the same finding one level up. Estimation broke for the same reason METR's follow-up study broke: the population being measured kept quietly changing what it submitted.

Where the afternoon actually went

Take a real-shaped ticket. Add a filtered export endpoint to a reporting service that has been in production for three years. Estimated at two days before agents, and honestly, two days was about right.

Here is the same ticket, split into what the work is actually made of:

PhaseBeforeNow
Understanding the problem and the code around it4h4h
Writing it6h1h
Reviewing it1h4h
Integration: the migration, the config, three call sites3h3h
Consequences: the bug that surfaces in week two2h3h
Total16h15h

Writing fell by five sixths. The total fell by an hour.

Three of those five rows did not move at all, because none of them are typing. Understanding a codebase is reading and holding, and an agent that reads it for you leaves you holding nothing. Integration is negotiating with systems that already exist and already have opinions. Consequences arrive on their own schedule.

The row that moved in the wrong direction is review, and it moved because the thing being reviewed changed. Reviewing code you wrote is remembering. Reviewing code you did not write is reconstruction, and it is the expensive half of the job now. Four hours to review one hour of generated code is the actual price rather than a failure of discipline, and the reason so many teams post an improved cycle time is that they are not paying it.

Your rows will differ. Mine differ by codebase, and the ratio moves a long way depending on whether I know the subsystem. What holds across all of them is the shape: the one row that collapsed is the row that was never the majority of the estimate.

What I estimate now

I estimate the verification, not the generation. The question I ask about a ticket is no longer "how long to build this" but "how long until I can defend it". Those were nearly the same number when I typed everything. They are not close any more, and the second one is the one that ships.

I estimate in unknowns, not in hours. How many things do I not know yet? Which of them can only be answered by touching production data, or by a conversation with someone who is on holiday? Generation did nothing to that list. It is the same list it was in 2019, and it still explains most of the variance in how long anything takes.

I do not re-estimate when the diff appears. This is the seductive one and the reason so many two-day tickets ship on day three. A complete-looking diff at 11am feels like evidence that the work is nearly done. It is evidence that one row of the table is nearly done. The unread diff is inventory rather than progress you can bank, and reviewing it is the part you had not started.

I record what I did, not what I would have done. Counterfactual self-report is the least reliable instrument in this whole discussion, and the survey says as much about itself. It puts the average overestimate of AI's effect on time at 40 percentage points, drawing on METR's 2025 trial, and warns that its own respondents are probably making the same error. That trial's headline number is now flagged as out of date. The perception gap is the part that survived.

So I stopped asking myself how much faster I was. I write down what the ticket took, and I compare it to the estimate I made before I opened the editor. That number is boring and it is measured, which is precisely what makes it worth having.

None of this makes estimates good. Estimates were never good. What changed is that the one part of the job that used to be predictable, the sitting-down-and-typing part, stopped being the part you were estimating, and a lot of teams updated their commitments as though it still was.

So when someone tells you their team got 3x faster, believe them. They are reporting uplift on new tasks, accurately, about a task list that no longer resembles the one your roadmap was built on.

Then ask what shipped.

(32 of 32)
01You don't have an AI problem. You have a process problem.02Why you should never ship code you don't understand03Stop copy-paste engineering04The lava layer: why AI code is slowly petrifying your codebase05The prompt is not the spec06The brilliant parrot problem: what AI actually does when it 'thinks'07The bureaucracy of bots: why we are checking the checker08The arms race for your trust: Mythos, Cyber and the security hype09Stop letting your agents write Markdown10You can't spot the bug if you didn't write the code11One in four: the security debt nobody's counting12Your 10x developer is gated by a 0.1x pipeline13Caveman vs context-mode: small mouth, or smaller room?14Code churn is the lava you can still measure15The ceiling is made of concrete16The token-saver tax: walking back my Caveman advice17Even the malware is AI slop now18ThePrimeagen was right19Tokenmaxxing is what happens when you measure the wrong thing20They just asked the bot nicely: your support agent is the attack surface21Speed got cheap. Judgement didn't.22The Ferrari has a limiter: a day with Claude Fable 523The off-switch was never yours24An open MCP server is worse than an open database25The off-switch works both ways now26AI writes the tests. Mutation testing checks if they work.27How to get better at reading code: a practice routine28Learning to program in the age of AI: what I would learn first29Who is responsible for AI-generated code? You are, and 2026 wrote it down30When not to use AI for coding: the tasks I still do by hand31Hiring junior developers in 2026: the collapse is a hiring decision32Software estimation with AI: the typing was never the estimate