Tokenmaxxing is what happens when you measure the wrong thing

In June I wrote this post from headlines and got several of them wrong. The argument held up. Four months later the companies involved have said so themselves, and I have measured a small version of it. This is the corrected piece.

What happened, sourced this time ​

The token as a productivity metric started at the top. On the All-In podcast during GTC in March, Jensen Huang said of his engineers: "If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed." The line about managers who discourage AI came earlier, from an Nvidia all-hands in November 2025: "Are you insane?" My June version merged the two.

Then the leaderboards. "Claudeonomics" was a dashboard a Meta employee built to rank colleagues by token use, and it came down after it was reported in April. I had attributed it to Amazon. Amazon had its own, KiroRank, where according to the Financial Times staff ran the tool on trivial tasks to inflate their counts. Amazon took it offline at the end of May, citing compute costs.

The bills followed. Uber spent its 2026 budget for AI coding tools in four months. Its COO, Andrew Macdonald, said on the Rapid Response podcast in May that the connection to results was missing: "That link is not there yet." I had given that line to the CTO, in words he never used. A consultant told Axios about a client that spent $500 million on Claude in one month without usage limits. That report has one unnamed source and nobody has confirmed it since.

What changed since June ​

The companies that pushed usage have started pulling back, and they say why.

• Microsoft. In early August an email from EVP Jay Parikh, reported by 404 Media and The Register, put it plainly: "Tokenmaxxing is not what we are optimizing for." It listed what was being paid for: "idle agents left running, questions asked of the model that the engineer already knew the answer to, work split into more turns than it needs." Divisions now get token targets. In June I wrote that Microsoft had cancelled most Claude Code licences. It was one division moving to Copilot CLI.
• Uber. On 7 August its CTO, Praveen Neppalli Naga, said: "We're coming to the end of the so-called tokenmaxxing era."
• The industry. In LeadDev's AI Impact Report 2026, 19% of respondents call tokenmaxxing effective and 57% say it fails to show real value.

None of this means the tools do nothing. A Microsoft field study from July found that engineers who adopted Claude Code or Copilot CLI merged roughly 24% more pull requests, with the authors noting that "a merged PR is not the same as the value it delivers". A second study, of 802 developers at a company with a mandate to double output, saw throughput reach 2.09 times the baseline, while the load per reviewer roughly doubled. The output is real, and the token count is a poor way to find it.

The same fix at five effort levels ​

Leaderboards measure people. I wanted to see the same thing inside a single task: does spending more tokens on a problem get you a better result?

Claude Code lets you set the effort level per session, from low to max, which controls how much the model thinks and how thoroughly it works. I gave Opus 5 one small bug fix at every level, three runs each, on Claude Code 2.1.273.

The task is a 45-line JavaScript LRU cache with a time-to-live, with several places where the code breaks its own README. The issue describes only one of them: a recently read entry gets evicted. The prompt was the same every time:

Fix the issue described in ISSUE.md. The intended behaviour of the cache is
specified in README.md. You can run the tests with npm test.

claude -p "$PROMPT" --model claude-opus-5 --effort $LEVEL \
  --tools Read Edit Write Bash Glob Grep --permission-mode acceptEdits \
  --setting-sources "" --strict-mcp-config --output-format json

Afterwards I graded each result with 13 tests the model never saw, covering the whole README. The original code passes 5 of them. A correct implementation passes all 13.

Fourteen of the fifteen runs fixed everything the hidden tests check. max produced 22 times the output tokens of low, cost 13 times as much and took 20 times as long, for exactly the same passing tests. On a tokens leaderboard, the max runs win by a distance.

The one lower score is worth reading. That xhigh run fixed the eviction bugs that belonged to the issue, and listed the remaining problems with expired entries, size and delete with an offer to fix them. That is a defensible call on a prompt that said "fix the issue", and it shows how little the test count says about judgement.

The limit of this test is its size. low already reaches the ceiling, so all it shows is that the extra tokens bought nothing here. Whether they buy more on larger, less clearly specified work is a question this test does not answer, which is why it pays to pick the effort level per task instead of defaulting to the highest. A larger study points the same way at scale: in an April analysis of eight frontier models on SWE-bench Verified, runs on the same task differed by up to 30 times in tokens, and "higher token usage does not translate into higher accuracy; instead, accuracy often peaks at intermediate cost and saturates at higher costs."

Goodhart, correctly quoted ​

The line everyone quotes, "when a measure becomes a target, it ceases to be a good measure", is Marilyn Strathern's 1997 summary of Charles Goodhart's 1975 point. My June version gave it to Goodhart. The idea holds either way, and tokens are an unusually clean case: the number rises whether or not anything useful happened, and the easiest way to raise it is to do less useful work more expensively.

You have seen the pattern with velocity points and code coverage. Tokens add something those never had: every point on the leaderboard arrives as an invoice, especially now that providers have switched the meter on.

What to count instead ​

What Microsoft's memo complains about and what the two field studies measured suggest a handful of numbers to watch. None of them is a token count.

• Cost per merged change. Token spend divided by what actually reached main. It rises when agents idle or loop, which is exactly the waste Parikh listed.
• Review load per reviewer. In the mandate study the load per reviewer roughly doubled along with throughput. If this rises faster than merged work, the bottleneck has moved to people, and more tokens make it worse.
• Revert and rework rate. How much of what merged was undone or rewritten within a few weeks. Throughput that comes back does not count.
• Time to a merged fix. From issue to merge, which is what the business was trying to buy.

These are also the only honest input to a plan, which is where the self-reported speedups fall apart once you check what shipped. And if none of them moves, the problem was probably the process all along.

The companies that built token leaderboards measured what was easy to count, and what was easy to count turned out to be the cost. My test says the same on a scale of a few dollars: the most expensive run was the one that tried hardest, and it fixed nothing the cheapest run had not.