METR set out to answer one question: does AI make experienced developers faster?
They could not.
Their February update explains why the follow-up experiment produced numbers they will not stand behind. Developers declined to take part, because taking part meant working without AI on half their tasks. Of those who did sign up, somewhere between 30 and 50% quietly filtered what they submitted, holding back the work they did not fancy doing by hand.
The study was built to measure whether AI helps. It was wrecked by developers deciding, task by task, where they wanted the help.
That is the interesting finding, and nobody quoted it.
The aggregate number stopped meaning anything
The headline everybody remembers is from the original trial: 16 experienced open-source developers, 246 real tasks in repositories they had worked in for years, and a 19% slowdown when AI was allowed. Those same developers believed they had been 20% faster.
Run it again a year later and the point estimate is 18% faster for the same group. Newly recruited developers came in at 4% faster, with a confidence interval sitting comfortably across zero. METR calls all of it very weak evidence.
So the number went from 19% slower to 18% faster in about a year, and the people who produced both numbers are telling you not to lean on either.
A quantity whose sign inverts that fast was never a property of the tool. It is a property of what you pointed the tool at. "Does AI make you faster" is the wrong resolution, like asking whether power tools make you a better carpenter without mentioning whether you are framing a wall or carving a dovetail.
The question that survives is per-task. Which is exactly the question METR's participants were answering, privately, when they refused to submit certain tickets.
What handing work over actually bought
DORA's data is clear that throughput moved. Epics completed per developer are up 66.2%. Release throughput no longer takes a hit from AI adoption, which was a genuine finding a year ago and is now table stakes.
The same data shows what came with it. Pull request size is up 51.3%. Median time in PR review is up 441%. And 31% more pull requests now merge with no review at all.
Generation got cheap. Everything downstream of generation got more expensive, and some of it got skipped. DORA's own framing is that AI is an amplifier, which is a polite way of saying it does not decide anything for you.
Difficulty is the wrong axis
Most developers sort tasks by how hard the task feels. Hard means "the agent will get it wrong", so hard work stays in-house and easy work goes over the wall.
That heuristic fails because model failures do not track human difficulty. A model will produce a working red-black tree and then get a two-line date comparison wrong, because one is well represented in its training data and the other depends on your codebase's particular idea of what a business day is. It has no world model telling it which of those two is the risky one.
This is the research that reordered my own thinking. A survey of operational failures in agentic code assistants catalogued 547 confirmed real-world incidents from the issue trackers of major code models and agent frameworks, across 33 risk types. The top failure category is constraint and instruction violation at 40.4%, with destructive operations second at 24.5%. Of all incidents, 60% were rated high or critical severity, and 170 involved data loss.
Then the part that matters for task selection. Bug fixing and setup or configuration work account for over 65% of all observed incidents, and they carry the worst severity profile in the set: 65.1% and 68.4% of those incidents were high or critical.
Bug fixing. The task everyone delegates first, because it feels bounded and verifiable and low-stakes.
It is the most dangerous thing on the list, and the mechanism behind that is simple. Bug fixing and configuration are the two categories where the agent must mutate state that already exists and that someone depends on. Greenfield code fails loudly in review. A bad migration, a rewritten config, a "fix" that changes behaviour three modules away: those fail quietly, in production, later.
The three questions I actually ask
I sort by exposure instead. Three questions:
Can I check it more cheaply than I could write it? This is the whole economic case, and the one people skip: if verifying the output takes longer than producing it, delegating costs me time and buys me a thing I understand less well. Boilerplate, a test suite against a contract I already defined, a migration of a known pattern across forty files: cheap to check. A caching layer with subtle invalidation rules: I have to reason through every path either way, so I may as well have written it and kept the reasoning.
Is it reversible? Everything the agent touches that I can throw away costs me nothing when it is wrong. Everything it touches that mutates real state costs me an incident. An agent wiped a production database on a box I owned, and the destructive-operations figure above says that story is common enough to have its own category.
Will I have to hold this in my head at 3am? If the answer is yes, I write it, or I read it line by line until I could have written it. This is the operational half of the ownership argument: you merged it, you own it settles who is accountable, and it leaves open the question of which work you should be signing your name to in the first place.
What I still do by hand
Novel design. Anything where the first correct decision is which shape the problem has. Models are excellent at the second, third and hundredth instance of a pattern and weak at the first. I will use one to argue against a design once I have one.
Subtle state. Concurrency, cache invalidation, transaction boundaries, anything with a partial-failure story. Agents write it beautifully, and that is the trap: the failure mode is a race that shows up once a fortnight, and I need the whole model in my head to recognise it.
The commit I would have to defend. Security-relevant code, money-handling code, anything I would be asked about in a post-mortem. I will let an agent review those. I will not let it author them unread.
Learning something. If the point of the task is that I come out of it knowing the subsystem, delegating it defeats the task. This is the one I have to enforce on myself, because it is always the cheapest thing to skip on a busy day.
Everything else, which is most of the work, goes to the agent inside the workflow I use: specced, contained, reviewed.
Write your line down
The METR participants who withheld tasks were doing the thing this whole post is about, using judgement they could not articulate, and it happened to land in a dataset where it looked like noise.
You have that judgement too. You applied it this week without noticing. The difference between having it and having it as a skill is whether you can say the rule out loud, defend it to a colleague, and notice when it turns out to be wrong.
Mine fits in a sentence. If it is cheap to verify, safe to undo, and I will not be asked to explain it under pressure, the agent gets it.
Everything else, I still type.