That happened to me dozens of times in one repository over the past month, often enough that I kept switching auto mode off to get anything done. I had added Agent to the allowlist in that project. It made no difference, and the reason is documented, just not anywhere you would think to look.

What auto mode gets right

Auto mode replaced the permission prompt with a second model. Since Claude Code 2.1.283 it is the starting mode for interactive sessions on every plan. Instead of you clicking yes, a classifier reads each action against a list of block rules and lets the routine ones through.

Mostly that is a clear improvement on the modes and allowlists I wrote about in July. Prompt fatigue is real, and a reviewer that never gets tired catches the force-push and the npm publish you would have waved through at the fortieth prompt. In the same repository it also stopped git commands it read as destructive and an attempt to create a public surface, which is exactly the class of action it exists for.

The trouble is subagents.

The rule that disappears

Claude Code decides every action in a fixed order. Your allow, ask and deny rules come first, read-only actions and edits inside the project come second, and everything else goes to the classifier. So the obvious fix for a blocked subagent is an allow rule for Agent, which should resolve at step one.

It doesn’t, because of a paragraph in the docs:

On entering auto mode, broad allow rules that grant arbitrary code execution are dropped

The list under it covers blanket Bash(*), wildcarded interpreters, package-manager run commands, and Agent allow rules. Claude Code puts them back when you leave auto mode. While you are in it, your "Agent" entry is gone, and nothing in the block message mentions it.

The reasoning holds up. A subagent can run anything its tools allow, so a blanket allow for Agent is the same hole as Bash(*). But the docs also say that the classifier checks a subagent at three points: the task description at spawn time, every action while it runs, and its report when it finishes. The spawn check is where my reviewers died.

A month of one repository, counted

Claude Code writes every session to a JSONL transcript, and a blocked call comes back as a tool result with the classifier’s reason in it. So I counted them across one repository, my Knossos MCP server, from 12 September to 14 October: 20 session transcripts on Claude Code 2.1.269 through 2.1.296. Four of them are forks of one conversation that carry copies of its history, so I counted every call once.

Almost all of the blocked launches were the plainest work there is: “Review Task 9 (spec + quality)”, “Re-review Task 8 fix round 1”, “Implement Phase 2 Task 5: release gate”. The repository runs its plans through subagent-driven development from the Superpowers plugin, so a single feature means a dozen subagents, most of them reviewers told to read a diff and report back without touching anything.

I looked for wording that set the classifier off and found little. Token, bypass and permission show up about as often in the prompts it allowed as in the ones it blocked. Push leans the other way, in 42% of blocked prompts against 33% of allowed ones. The word nudges the odds a little and decides nothing.

The retries settle it: 26 times a blocked prompt was sent again word for word, before the session’s fallback kicked in, and 24 of those went through. Same input, different verdict. The Auto-Mode Bypass rule exists to catch an agent “taking a denial in bad faith and tunneling the same action through a different path”, and the identical retry is the one path it reliably let through.

Then there is the cap. After three blocks in a row or twenty in a session, auto mode pauses and falls back to prompting, and those thresholds are not configurable. Two of my sessions hit 36 and 31 classifier blocks in total. That is the moment I kept switching auto mode off by hand.

One smaller finding: three Agent calls got no verdict at all, because one of my own hooks rewrote the call’s input after the model wrote it, so the classifier was asked to judge something other than what would run. Claude Code refuses those, which is correct, and it reads exactly like a block.

What actually helps

Write your workflow into autoMode.allow. The classifier’s own exceptions live in autoMode.allow, a list of prose rules, and Auto-Mode Bypass is a soft block, so an allow entry can override it. This is the one I added to ~/.claude/settings.json, with "$defaults" kept so the built-in exceptions stay:

{
  "autoMode": {
    "allow": [
      "$defaults",
      "Dispatching subagents with the Agent tool to implement, review, re-review or research code in the current working repository, including instructing an implementer to edit files, run the test suite and commit to the current branch, is allowed: it is the user's standard plan-execution workflow."
    ]
  }
}

Describe the setup too. An autoMode.environment entry that says reviewers are read-only and implementers stay in the current repository gives the classifier the context it otherwise guesses at. Run claude auto-mode critique on the result, and claude auto-mode config to see what the classifier actually reads. /auto-mode-setup drafts these entries for you and offers to remove the allow rules in ~/.claude/settings.json that auto mode ignores anyway.

My own numbers do not show it closing the gap. I added these entries somewhere between 6 and 13 October, and on 13 October three of fourteen Agent calls were still blocked. Two of those were Data Exfiltration, which is a hard block: no allow entry and no user intent overrides it, by design.

Use /permissions instead of switching modes. The Recently denied tab lists every block, and pressing r lets the model retry that one call. Given the retry numbers above, that clears most false positives without leaving auto mode.

Don’t reach for the bypass. A mod that handles tool.check can approve an action before the classifier sees it, and the classifier never checks what the mod approved. That works, and it also means writing your own Agent(*) with extra steps. A subagent’s actions still go through the classifier while it runs, but the spawn check exists for a reason.

Plan subagent-heavy work in Manual mode. If a session is going to fire forty reviewers, one Shift+Tab and an approval per launch is faster than hitting the twenty-block fallback halfway through.

A fixed allowlist is boring and predictable, and the classifier is neither. It reads my reviewer prompt, says no, then reads the same prompt again and says yes. That is a dice roll at the spawn point, and I would rather know which rule decides than how often it lands my way. Until that check gives the same verdict on the same input, treat a blocked subagent as noise first, and a subagent with a narrow tools allowlist as the thing that actually limits what it can do.

That is the detail worth keeping from the Bricksforge story back in the news today. It is tempting to file an upload vulnerability under missing validation and move on. This one had validation. The next step stopped respecting what it established.

I care about that distinction because it changes what I would look for in a review. Finding a MIME check in an upload handler answers one question. Following the file until the application uses it answers the one that matters.

Where the checked image became PHP

Patchstack’s report, published on 8 October, describes CVE-2026-85097. An unauthenticated attacker could obtain a nonce, upload a GIF containing PHP code, and submit a form with altered metadata.

The file passed the initial MIME check. During submission, the file field referred to that image, while the client-controlled url field named a PHP destination. Bricksforge trusted the metadata and placed the contents there. Requesting that PHP file then executed the embedded code.

The same bytes went through two different interpretations. The upload check recognised an image. The eventual destination let the server treat its contents as a program.

Here is the failure compressed to the two fields that matter. This is a schematic, not a complete request:

file: /uploads/tmp/image.gif
url:  https://example.com/uploads/image.php

The dangerous decision happened at the handoff: information sent back by the browser determined where the server put the file.

A check has a scope

An image check is useful. It can reject input your application does not support, catch mistakes, and contribute to a layered upload policy. I would keep it.

But what exactly did it prove?

It established something about the uploaded contents at that point in the workflow. It did not authorise a later filename, approve a destination, or bind a subsequent form submission to the server’s original record of the upload.

Those are separate decisions. Each needs its own source of authority.

The mistake is assuming a validated file stays safe through every later step. A file may acquire a different extension. A client may return different metadata. A later handler may interpret a field as an instruction that the first handler treated as display data.

In a review, I want to find where that interpretation changes. Who supplied the value? Who can replace it? What decision does it control now?

That is also why a screenshot of a tidy upload widget tells you little about the server behind it. The Bricksforge changelog for 4.0.1, dated 8 October, says the issue affected every site with Pro Forms active, including forms with no upload fields. The visible form was a poor way to judge exposure.

Read across the handlers

Reviewing the upload function on its own encourages a comforting little inventory: size limit, type check, temporary directory. Everything looks sensible.

Then you open the submission handler. It has a file reference, a URL, some metadata. Those names look reasonable too. The problem appears when you trace which values survive between them and which ones the client can replace.

This is the kind of reading I want developers to practise. In my routine for reading unfamiliar code, the useful exercise is predicting what the code will actually do. For an upload pipeline, extend that prediction across the request boundary.

Write down the final path before running anything. Explain where every part of it came from. If the answer includes a client field, ask why that field gets to make a storage decision.

An agent can help find the callers and follow the data. The judgement still has to be yours. Ask it to trace each value into the final write, then check the trace against the code.

If you stop at a fluent explanation of the upload validator, the rest of the workflow stays unread. That is adopting code without understanding its decisions.

I have no evidence that AI wrote the vulnerable Bricksforge code. This is an ordinary software boundary that deserves a better review.

Keep the storage decision on the server

OWASP’s file-upload guidance recommends application-generated filenames, restricted allowed types, and storage outside the webroot or on a separate host. It also warns that a file-signature check cannot carry the defence alone.

For a two-step upload flow, my design would keep the server’s record authoritative throughout. The first request creates a record for the stored object. The later request supplies an opaque upload ID. The server looks up the path, permitted type, and ownership or submission binding itself.

An opaque ID still needs authorisation. A random-looking identifier is no reason to let one submission claim another submission’s upload.

The original filename can remain as display information. It should not become a filesystem path simply because the browser sent it back. Likewise, a public download URL should be derived from the stored object, rather than accepted as instructions for where to write it.

I would also keep uploaded content out of executable locations. Where the application must serve it publicly, the server configuration needs to treat it as data, with no PHP handler for that storage area. That gives you another boundary if an application-level check fails.

These are design recommendations, not a description of Bricksforge’s patch. The vendor’s changelog confirms the fix without documenting its implementation.

Test the journey the attacker takes

A unit test showing that the MIME validator accepts a GIF can be correct while the complete system remains vulnerable. The useful regression test continues past that point.

Start with a permitted upload. Complete the later submission with metadata that disagrees with the original object. Then inspect what was actually stored and served.

I would cover these cases:

• Changed destination. Submit an executable extension or a path outside the intended storage area. The client must never choose the final write location.
• Changed reference. Try another submission’s upload ID, or reuse one after it has been consumed. Check the ownership and lifecycle rules your application actually promises.
• Unexpected upload data. Send upload metadata to a form that has no upload field. The backend should enforce its configured fields rather than trust the browser’s form layout.
• Execution at retrieval. Request the stored file through the deployed serving configuration. Verify that uploaded contents cannot run as server-side code.

Those tests check behaviour across upload, submission, and retrieval. They make the security property explicit: client-supplied metadata cannot turn an accepted upload into an executable file.

They belong in an isolated test environment, with fixtures you control. You do not need to reproduce an attack against somebody else’s site to check your own boundary.

The patch closes the entry point

Patchstack identifies 3.1.8.10 as the fixed version for the reported 3.x range. The vendor lists the Pro Forms upload fix in 4.0.1. Check the release line you operate rather than treating a larger version number as proof that a fix is included.

Patchstack also reports exploitation attempts beginning on 7 October at 21:47 UTC. On a site that was exposed, it recommends checking historical logs and the host for unexpected PHP files or unauthorised changes.

Patching and checking for an earlier intrusion answer different questions. Closing the upload route says nothing about a file an attacker already left behind.

The part I would take back to my own codebase is the handoff. Find the next handler after the check. Trace the values that determine where the file is written, then check how the server serves it.

The validator did its job. The workflow needed one too.

I have been programming since 1984. In that time I have watched a fair number of technologies arrive that were going to change our work forever.

Some of them did.

We went from assembly and BASIC to higher-level languages. From hand-written HTML to frameworks. From our own servers to the cloud. From Stack Overflow to an agent that walks half your codebase before your coffee goes cold.

And at every step, the same thing was said: developers are about to become redundant.

It never happened.

This time something is different, though. AI has not suddenly become intelligent, and Claude and Codex still write software with bugs in it. The difference is economic.

You don’t have to be replaced to lose your job

Say you have a team of twenty developers.

In two years, ten of them might ship the same amount of software. Those ten are the same developers, with agents taking over most of the execution.

Then the company has a choice.

Keep twenty people and build twice as much. Or keep ten and keep delivering the same.

Which one do you think the CFO is going to pick?

That is no longer a thought experiment. In February, Jack Dorsey’s Block went from over 10,000 employees to fewer than 6,000. Dorsey gave the reason himself: “We’re already seeing that the intelligence tools we’re creating and using, paired with smaller and flatter teams, are enabling a new way of working which fundamentally changes what it means to build and run a company.”

TechCrunch keeps a running list of the big tech companies that named AI in a round of layoffs this year. By the end of July it had 21 on it.

Of course this is a simplification. Software development is more than writing code. Requirements are vague, systems are full of historical decisions, and production has the annoying habit of ignoring your benchmarks.

Some of those layoffs are also plain cost-cutting with a more modern story attached.

But the argument that AI cannot replace a complete developer misses the point.

It doesn’t need to.

Demand is growing again, just not for everyone

Since this summer, the counter-argument has numbers too.

According to Indeed Hiring Lab, US software development postings grew almost 15% after Claude Code launched in late February 2025, while total postings fell 7% over the same period. After years of contraction, the occupations most exposed to AI are now rebounding the hardest. Indeed is careful to say correlation is not causation, but calls the timing “a coincidence that cannot be ignored”.

That is good news, and I am not going to explain it away.

Look at who is being hired, though. Of the growth between May 2025 and May 2026, 71% came from senior roles and 37% from jobs with AI in the title. Indeed’s own reading: “demand is growing for experienced professionals who can work with AI, not necessarily a broad-based recovery across all software roles.”

So the market is growing and shrinking at the same time. Block cuts its headcount, the job boards fill up again, and which side you land on depends on who you are.

The junior goes first

Every senior developer spent years fixing bugs, making the wrong architecture calls, digging through production incidents, and learning why that one elegant solution turned out to be a disaster in practice.

You don’t learn that from a prompt.

But if an agent handles the simple tickets faster than a junior, why would a company still hire the junior? That 71% is the answer the market is giving right now. I wrote earlier about the seed corn the profession is eating: if nobody trains juniors, where do the seniors of ten years from now come from? And the one large employer still hiring juniors shows what the role has to become for that investment to pay off.

The short version: experience has to be built somewhere. And the work you used to build it with is exactly the work that is easiest to automate.

The senior is not safe

I regularly hear experienced developers say that AI mostly takes over the simple tasks.

Architecture, complex debugging, security and system design stay human work.

For now, a lot of that holds. The postings data says the same.

But it is dangerous to turn a temporary limitation into a permanent protection.

Today’s models will look dated by 2028. And once agents can investigate a problem on their own, design a solution, run the tests and fix their own mistakes, the line moves again.

The senior postings point the same way. They ask for people who can work with AI: someone who directs agents, reviews their work and carries the responsibility. That is the role where you have to ask how many of them are needed.

Maybe there will always need to be a human who is accountable.

But how many?

One per five agents? One per twenty? One engineer overseeing an entire platform?

Nobody knows.

And anyone who claims with certainty that architects and seniors are safe is doing what the AI vendors do when they promise everything will run autonomously next year.

They are selling certainty they don’t have.

More software does not automatically mean more jobs

The usual counter-argument is that cheaper software development creates more demand for software.

It might.

When computers got cheaper, we didn’t use fewer of them. We found thousands of new uses. In June I called that Jevons paradox wearing a hoodie, and I still believe it holds for the demand for software.

But there is an assumption buried in that story: that extra demand for software also produces extra demand for human developers.

Those are two different things.

If AI drives production costs down far enough, the amount of software can explode while the number of people needed to make it falls.

More code. More applications. More systems.

Fewer developers.

Nothing in that combination contradicts itself. The Indeed numbers don’t refute it either. A posting is not a job, a rebound from a trough is not long-term growth, and a market that mostly wants seniors is not growing in breadth.

My guess: 2030 gets uncomfortable

I don’t expect every developer to be out of work by 2030.

What I do expect by then is a much clearer split between people who produce software and people who carry responsibility for what software does.

The first group will have a hard time.

The second will stay necessary for now, probably in smaller teams.

And somewhere between 2030 and 2035, the traditional picture of a software team might well disappear for good.

That is my estimate of where the technology and the economic incentives are likely to meet, and no reliable labour-market formula backs it up.

I could be five years off. Maybe ten.

Maybe demand for software turns out to be so large that we need more developers rather than fewer.

But I would not bet my career on that last scenario.

The uncomfortable answer

I work with AI agents every day. I see how impressive they are, and how many mistakes they still make.

I see code that appears in minutes and takes hours to review properly. I see duplication, missing context, and solutions that are correct locally but wrong at the level of the system.

Those are real limitations, and none of them protects your job.

If you want to be in the second group, make sure your work consists of what an agent cannot carry. Own a system all the way into production, and review what gets generated instead of passing it along. Be the person who can explain why something was built the way it was, because whoever merges the code answers for it, no matter who wrote it. And understand why judgement became the scarce part once speed got cheap.

For now, that is the side of the line where demand is growing.

The question is not when AI can do everything a developer can.

The question is when AI can do enough for a company to decide it can carry on with fewer developers.

And that moment may come a lot sooner.

We spent years building software to automate other people’s work.

Now it’s our turn.

All three work. I ran the same task forty times, in two sizes, to find out what the timing costs, and every run passed. What differed was the bill: on the bigger task, correcting after the agent finished cost a median 56% more than saying it up front. Stopping it with Esc at its first edit cost nothing extra there.

The experiment

A small Node project with three CLIs reading the same sales CSV. The task: add --since and --until date options to all three. Then the correction you always think of a moment too late: put the filtering in one pure function in lib/filter.js with its own unit tests, reject impossible dates like 2026-02-30 with exit code 2, and document both flags in the README.

Four routes, five runs each, identical text every time:

• Brief: task and correction in the first message.
• Esc: task alone, interrupted at the agent’s first file edit, then the correction.
• Queue: task alone, correction queued at the first file edit, no interrupt.
• Late: task alone, correction after the agent reports it is done.

Claude Code 2.1.293 on Opus 5.5, headless, with no user settings, hooks or CLAUDE.md loaded. A script scored each result on twelve checks. I ran a one-CLI version of the same task first, scored on seven. It left the agent little room to wander, which is why the three-CLI version exists.

Medians. All forty runs passed every check. The agent always got there. It just did more work on the way, and the later the correction, the more it had to undo: by the time the late route heard the real requirements, it had made six to eight edits that the correction turned into rework.

These are cents on a task that takes a minute. That is the point of the second column, though: going from one CLI to three, the late penalty more than doubled while Esc dropped to zero. Your real tasks are longer than mine.

Before it starts: give it the context you have

The cheapest correction is the one you never need to send. Most of what I put in the correction was knowable before the first prompt, and on the multi-file task every route that started without it made more edits.

Two things make the brief better without making it longer:

• @ a file to put its full contents in the conversation. @src/lib/filter.ts beats “the filter helper” every time. On a directory you get only a listing.
• Paste a screenshot with Ctrl+V (Alt+V on Windows and WSL, Cmd+V in iTerm2), drag it in, or give its path. A layout bug described in words costs a round trip that the picture skips.

The research agrees, from the other side. In Ambig-SWE, agents that asked clarifying questions on underspecified tasks performed up to 74% better than agents that just started. Plan mode is the built-in version of that: the agent shows you its approach before it writes anything, which is the cheapest moment to object.

Esc: stop, keep the work, redirect

The interactive mode docs describe Esc precisely: “Stop the current response or tool call mid-turn so you can redirect. Claude keeps the work done so far.”

That second sentence is why it cost so little in my runs. The agent keeps what it has: it has read the code, it knows the CLIs, and it has one edit it now has to revisit. You lose only the direction it was heading in, which was the wrong one anyway.

The habit to build is pressing it at the first sign, not the third. Anthropic’s best practices put it as “Correct Claude as soon as you notice it going off track”, and the reason shows up in the research on how agents fail. Failure as a Process traced failed runs of CLI coding agents on Terminal-Bench, with no human in the loop, and found that “half of all failed trajectories have already committed their decisive error by step 7, but do not become unrecoverable until around step 12, and do not produce externally observable evidence until around step 16.”

Between step 7 and step 16 the agent is still working, and it looks like it is going fine. That gap is where Esc pays off.

Queue a message when it is close enough

You can also talk to the agent without stopping it. Type while it works and press Enter. Per the docs, “Claude Code passes it to Claude as soon as those tool calls finish, within the same turn.”

That is softer than Esc, and in my runs it cost 15% more than the brief on both tasks, because the agent finishes the batch of tool calls it is in before it reads you. Fine when the correction is a detail. When the direction is wrong, every tool call it completes first is work you are paying for and then paying to undo.

Two keys make the queue more useful than it looks. Ctrl+Enter sends queued messages right away. And Up from the first line of the input box takes your queued messages back so you can edit them before they go out.

One catch: a message queued mid-turn gets no checkpoint of its own. To undo what Claude did after it, you rewind to the prompt that started the turn.

/btw for the question that is not a correction

Halfway through a task you wonder why it picked a library, or which file handles the auth. Asking in the main conversation puts the question and the answer into the context for the rest of the session.

/btw is for exactly that. The docs: “Use /btw to ask a question about your current work without adding to the conversation history.” It sees everything so far, has no tools, and runs while Claude keeps working. If the answer needs tools after all, press f to fork it into a background subagent.

When to stop correcting and start over

Steering has a limit. Each correction leaves the failed attempt in the context, and models get more likely to make mistakes when their earlier errors are in front of them. Anthropic’s best practices draw the line: “If you’ve corrected Claude more than twice on the same issue in one session, the context is cluttered with failed approaches.”

At that point you have two exits. Esc Esc opens the rewind menu, where you return to the prompt before the detour and send it again with what you have learned. Or you clear the session and start with a brief that includes the correction from the start, which was the cheapest route in my experiment, or tied for it.

Steering is the skill

Anthropic studied around 400,000 Claude Code sessions for its June report on expertise and agentic coding. One of its conclusions: “Part of the value of expertise appears to be the ability to steer the agent in the right direction.” Among the users it rated as novices, 19% of sessions ended abandoned, against 5 to 7% for everyone else.

Experienced users also interrupt more often. An earlier Anthropic study found new users interrupting in 5% of turns and more experienced users in around 9%. They know when a turn is heading somewhere expensive, and they say so while it is still cheap.

That is all steering is. The agent will get there either way. You decide how much of the trip you pay for twice.

That was almost seven weeks ago. Here is the number. It surprised me.

I assumed the agent’s time went into thinking. A big model, long reasoning, a spinner that turns because something hard is happening. Over a month of my own sessions, the main agent spent 74 hours generating and 117 hours waiting on its tools. And part of that waiting was the agent sitting idle while I was somewhere else.

How I measured it

Claude Code writes every session to a JSONL transcript, and every record in it carries a timestamp. That is enough to time almost everything.

• A tool call runs from the record where the model asks for the tool to the record where the result comes back.
• Model time runs from each input (my prompt, or a tool result) to the last thing the model writes before it calls the next tool, cut at the end of each turn so idle gaps between turns do not count.
• Subagents keep their own transcripts, which I measured separately.

The transcripts on my machine go back to 4 September. Through 4 October that is 127 sessions, 1,804 prompts from me, 3,503 turns and 1,106 subagent runs, on Claude Code 2.1.260 through 2.1.289, almost all of it in auto mode.

One limit matters for everything below. The clock on a tool call includes any approval prompt in front of it, and the transcript does not record the prompt. So when a call “took” twenty minutes, some of those minutes may have been the agent waiting for me to click yes. I come back to that.

Most of the time is in the tools

The model’s own steps are quick: a median of 5.5 seconds between getting a result and asking for the next tool.

The hours add up elsewhere. Across the main agent in the month:

The tool rows add up to 125 hours. Calls that ran in parallel overlap, so on the wall clock the agent spent 117 hours waiting on tools.

Inside the subagents the share is higher still: Bash took 48 of their 55 tool hours, 87%.

Of everything the agent waits on, the shell weighs the most.

Half of all shell time is 1.4% of the calls

Across main agent and subagents together, Bash ran 46,574 times in the foreground, for 117 hours. The median call took 1.3 seconds, and nine in ten finished within 9.8 seconds.

Then the tail. The slowest 674 calls, 1.4% of them, hold half of all Bash time. The 1,420 calls that ran longer than a minute hold 76 hours, almost two-thirds.

Sorting those hours by what the command did:

• Tests: 22 hours, a fifth. No surprise and no regret, because a suite that takes time is doing its job, and an agent that runs it is the point of the exercise. In turns that ran tests the median was two runs, and the record was 77 in a single 69-minute turn: a fix loop on one of my own projects.
• sleep and polling loops: 12.6 hours. Most of that is the agent writing sleep 60 or a while loop around a status check, and blocking the whole conversation until it returns.
• Reading and searching (cat, grep, find): 19 hours, scripts 16, git 11, for loops 10, builds and type checks 3.
• A fifth I could not classify: compound one-liners that cd, set a variable, loop and pipe in one call.

The sleep hours bother me most, because the fix already exists. Claude Code can run a command in the background and tell the agent when it finishes. In the month, 690 calls used that. 1,420 calls blocked the foreground for more than a minute.

Failures barely register

1,277 tool calls failed in the month. The largest group, 518, were shell commands that exited non-zero. Together the failed calls took 8 hours. A failure is loud in the transcript and quiet on the clock.

A few long turns hold the hours

The turns have the same long tail as the shell calls. 1,892 of them, more than half, were over within a minute, and together they make up 5% of all turn time.

At the other end, 39 turns ran for more than an hour. Those 39 hold 38% of the clock. Everything over a quarter of an hour, 205 turns, is two-thirds of it.

What working with the agent feels like and what it costs in hours are two different things. Most of what I see is a quick exchange. Most of what the clock records is a few long stretches.

The slowest tool is me

The agent spent 29 hours waiting on AskUserQuestion, the tool it uses to put a multiple-choice question in front of me.

It asked 468 questions in 382 calls. The median answer came in 53 seconds, and 205 calls were answered within a minute. When I am at my desk, a question costs less than a minute.

When I am not, it costs hours. Two questions waited more than an hour each, 9.4 hours between them. Nobody was there to answer them.

The same thing hides inside the tool clock. An Edit takes milliseconds, yet 42 edits and writes “took” more than a minute, 1.9 hours in total. Those minutes are almost certainly approval prompts: an edit to a file a hook guards, a write outside the project.

That 1.9 hours is a floor. A Bash command that waited on approval looks exactly like a Bash command that ran for a long time, and with 117 hours of shell time I cannot separate the two. Any “Bash is most of your tool time” number, my own plugin’s included, carries some of my coffee breaks in it.

Two sessions hide some of the wait

The main agents ran 283 hours of turns, and those fit into 231 hours of wall clock. For 44 of those hours, two or more sessions were working at once.

That is the counterweight to everything above. For about a fifth of the busy time more than one agent was at work, so a wait in one session did not leave everything standing still.

What I changed

Every fix comes down to the same thing: stop letting the waits block the conversation.

• Long commands go to the background. A test suite, a build or a remote run gets run_in_background, and the agent does something else or ends its turn. One line in CLAUDE.md asks for it. If asking is not enough, that is what hooks are for.
• No sleep loops. Waiting for CI or a deploy belongs to a background watcher that tells the agent when something changed. A foreground loop freezes the whole conversation.
• Questions before I leave. If the agent will need a decision, it should ask while I am still there. “Ask everything you need now, then work unattended” is a perfectly good prompt.
• A second session for the long waits. Overlap already absorbed 52 hours of turn time this month. A worktree per session keeps two agents out of each other’s files.
• Fewer prompts for the safe things. Every approval prompt is a place the agent can stall. The permissions guide covers allowing what is safe so that the prompts left are the ones worth stopping for.

None of this says the agent helped or did not help. A twenty-minute test run that caught a regression was twenty good minutes. Turn these numbers into a target and you are back to measuring the wrong thing.

What they did tell me is where the afternoon went. Less into thinking than I imagined, a lot into a shell that waited on a test suite, and a surprising amount into waiting for me.

One question stayed with me. Design gets screenshotted and passed around. Code mostly does not. If an agent fills every gap in a design brief with the median, what does it put in the gaps of a code brief, where nobody is looking?

So I measured it.

How I measured it

The idea was to give every model the same open brief and then count what it filled in. That sounds simple, and most of the work went into keeping it fair.

Three tasks with the gaps left open

Three small TypeScript tasks, each a few sentences long: a REST API for bookmarks, a CSV-to-JSON command-line tool and a client library for a flaky, rate-limited HTTP API. Each prompt names TypeScript and ends with “Include tests.” Everything else is left open on purpose: framework, storage, test runner, argument parser, retry strategy. This is the whole API prompt:

Build a small REST API in TypeScript for storing bookmarks. A bookmark has a URL, a title
and optional tags. Support creating, listing, fetching one, updating and deleting
bookmarks, and persist them so they survive a restart. Include tests.

TypeScript is fixed because language choice is a known bias of its own. Left open, the first Gemini run I tried wrote Python.

Every agent as shipped, in a clean room

Five harnesses: Claude Code, Codex, Antigravity, OpenCode and Freebuff. 23 models in total, each at the settings its harness ships with. Antigravity puts the reasoning level in the model name, so there I had to pick one: medium, and low for the Pro model, which has no medium.

Each run got a fresh, empty directory and a fresh process, and nothing else. The agents normally read a lot from your home directory: config, rules, memory, plugins, MCP servers. So every harness ran from a clean home that held only its login. Without that, Antigravity loaded my own MCP servers and Claude Code my Gmail and calendar connectors, and the runs would have measured my setup instead of the model.

Four of the harnesses run headless and report in JSON. Freebuff only has a terminal interface, so a small driver types the prompt in tmux, reads the screen and waits until the agent is done. When an agent asked a question before building, it got the same answer every time: “Choose whatever you think is best and build it.” Every run had 45 minutes.

A colleague (thanks, Jelle!) ran GPT-6 Astra and GPT-6.1 Sol on their own account, with the same harness and the same prompts, and sent back the results.

Checking which model actually answered

The name in a model picker is a claim. For every run I read the model that actually answered from the agent’s own transcript, session file or database, and flagged any run where the two disagreed. That turned out to matter.

Does it work, and what did it choose

Every project then went through a smoke test, the way a person would try it: install it with the package manager it chose, build it if its start script needs a build, start it and call the API, convert a CSV with and without a header, run the library’s own tests and type-check it with its own TypeScript. When a project failed, I read why. Several times the fault was in my test, and I fixed the test and reran it for every project.

I read the choices themselves from the source code alone, leaving the tests and the README aside: which framework is imported, where the data goes, which test runner is installed, how the retries back off. Each of those is an axis with a label per run. Whenever a rule mislabelled a real run, that run became a test case before the rule was fixed, and every run was labelled again.

Finally, the comparison. For each axis I took every pair of runs and counted how often the two made the same choice: pairs from the same model, and pairs from two different models. If a model only follows the trade, both numbers come out the same. If a model has its own habit, its own runs agree more often.

All 23 models

Per harness, each model with the runs that worked and the median time per run, in minutes and seconds:

• Claude Code: Haiku 4.5 (14/15, 2:51), Sonnet 5.5 (15/15, 0:51), Opus 5.5 (9/9, 2:04)
• Codex: GPT-5.6 Luna (11/15, 2:23), GPT-5.6 Terra (15/15, 1:42), GPT-5.6 Sol (8/9, 6:51), GPT-6 Astra (9/9, 3:47), GPT-6.1 Sol (9/9, 4:33)
• Antigravity: Gemini 3.8 Flash (15/15, 3:34), Gemini 3.7 Flash (15/15, 1:59), Gemini 3.6 Flash (9/9, 7:13), Gemini 3.1 Pro (3/9, 1:06)
• OpenCode: Big Pickle (9/9, 20:05), Muse Spark 1.3 (8/9, 2:33), Nemotron 3 Ultra (8/9, 9:12), Nemotron 3.5 Lightning (2/9, 22:23)
• Freebuff: GLM 5.3 Flash (9/9, 19:13), Solar Mini 4 (2/9, 39:05), MiMo 2.6 Flash (9/9, 8:22), Solar Pro 4 (3/9, 6:01), DeepSeek V4.1 Flash (15/15, 5:00), GPT-6 Luna (8/9, 2:57), MiMo 2.6 Pro (8/8, 7:19)

A run worked when it passed the smoke test. The chart above puts time next to tokens for the four harnesses that report them: input plus output for a whole run, mostly cached context that the harness reads again on every step. Freebuff only shows the size of its context.

Every harness ran at its own default reasoning level. In Freebuff that means max for GLM, high for DeepSeek and GPT-6 Luna, and none for the Solar and MiMo models. Only in Antigravity did I pick one, as described above.

The runs took place between 28 September and 1 October 2026, on Claude Code 2.1.285, Codex 0.154.0, Antigravity 1.2.13, OpenCode 1.18.29 and Freebuff 0.1.4. Seven models got five runs per task, the rest three. MiMo 2.6 Pro has one library run fewer: its paid hour ran out before the last run could start.

Four models are missing from this overview. Ling 3.0 Flash and the free mimo-v2.5 in OpenCode returned only errors, in all nine runs each. Space Bunny Alpha in Freebuff failed twelve times out of twelve. Muse Spark 1.2 turned out to be answered by a different model on Freebuff’s free plan, so there was nothing of its own to measure.

The whole experiment was cheap: about $13 at list price for all Claude runs, six points of a weekly Codex limit, Freebuff’s free daily allowance, and nothing for Antigravity or the free models in OpenCode.

18 runs did not count: the provider returned errors and left nothing behind. The rest is 248 runs.

The field nobody asked for

The prompt says a bookmark has a URL, a title and optional tags.

In 69 of the 83 API runs, the agent added createdAt or updatedAt anyway. 19 of the 23 models did it, nearly all of them in every run. Further down the same list: filtering by tag in 32 runs, a health endpoint in 27, a graceful shutdown on SIGTERM in 27. Nobody asked for any of these.

Each of these can be a good idea. If you want to know when a bookmark was saved, createdAt is exactly right, and an agent that adds it saves you a step. A health endpoint belongs in anything that runs behind a load balancer.

The trouble is that here nobody decided. The field is there because bookmark APIs usually have one, and it looks like diligence, so it gets through review. Once it is in, it stays: someone sorts on it, migrates it and argues about what it means. Created according to which clock? Does editing the tags count as an update?

That is the code version of a UI tell. Whether it belongs is your call, as long as you know it is there.

Who left the timestamps out is the interesting part. GPT-6 Astra and GPT-6.1 Sol: 0 of 6 runs. The three GPT-5.6 models from the same vendor: 13 of 13.

The newer two still add plenty: all six of their runs include a SIGTERM handler. They kept the operational extra and dropped the field in the data model, and the field is the one that is hard to take back. Gemini 3.1 Pro and Nemotron 3.5 Lightning skipped the timestamps as well, but both wrote very little code overall, so that says little.

The layer every model shares

Below the additions sits a layer that hardly varies at all:

• Strict TypeScript in 245 of 248 runs.
• No linter in 243 of 248. Five runs set one up.
• A hand-written retry loop in 79 of 82 library runs, with no retry package, usually with exponential backoff and plain fetch.

And yes, three retries, because that is what everyone uses. These are reasonable choices, the trade’s median.

Research on models outside an agent points the same way. A study of eight models found they prioritise familiarity and popularity over suitability when they pick a library or a language. LangChoiceBench found that when models pick Python, the choice is mostly “automatic or driven primarily by ease”. I fixed the language, so the ease shows up in everything else.

The missing linter is the one I would act on. Nobody decided against it either.

Every model line has a handwriting

Where the ecosystem itself is split, runs of the same model agree with each other far more often than runs of different models: a median of 17 percentage points across the axes I measured. To check that this was not luck from three runs, I gave seven of the models five runs per task.

It held:

Storage shows it most clearly. 60 of the 83 API runs keep bookmarks in a JSON file. SQLite comes almost entirely from three model lines: Gemini in 15 of its 16 runs, Opus in 3 of 3 and Big Pickle in 3 of 3, plus one run of GPT-6 Astra.

Haiku and Sonnet, in the same harness as Opus, chose JSON files. So that preference belongs to the model, whatever tool drives it.

Luna’s habit survived a version bump. The stack you get depends on which model happened to be selected, and a new model can change it without anyone changing a line of the brief.

Who asked first

Only Freebuff lets an agent stop and ask before it builds. The other four harnesses ran headless, where nobody is there to answer. In Freebuff, four of the seven models asked at least once: DeepSeek in 6 of its 15 runs, MiMo 2.6 Pro in all three of its API runs.

They asked about exactly the gaps I had left open: which framework, how to store the bookmarks, which test runner, how to stay under the rate limit. And the options often came with a favourite already marked. MiMo offered “Express + JSON file (Recommended)”, the two most common choices in the whole experiment.

So the question was more of a confirmation. Every one of them got the same reply, “Choose whatever you think is best and build it.”, and what they built next still counts as their own default.

What the name does not tell you

The picker and the transcript did not always agree. Freebuff’s “MiMo 2.6 Flash” answered as mimo-v2.5 in every transcript. Muse Spark 1.2, on the free plan, was answered by a different model altogether. Two free models in OpenCode never returned a working response all night.

And then there was GPT-5.6 Luna. In 2 of its first 9 runs it wrote five files outside its own working directory, noticed, and ran find across my whole experiments folder to locate one of them. It saw file names and read nothing else, but it is the only model in 266 runs that left its directory. None of the agents had approval prompts in the way, so nothing stopped it.

What this does not show

• Model or harness. I tried to run the same model in two harnesses and could not: the free MiMo endpoint returned only server errors, and Gemini CLI no longer serves the free tier. What I have are contrasts inside one harness, Opus against Haiku and Sonnet, and two versions of Luna in two harnesses.
• Quality. The smoke test checks that the code works, not that it is good.
• Anything general. Three small tasks, two to five runs per model per task, counts and no significance tests, all in an empty directory. A study of 26,760 pull requests written by agents found they rarely add a dependency, in 1.3% of PRs, and draw on a diverse set of libraries. Real projects look different.

My own harness was not flawless either. OpenCode takes its project directory from $PWD rather than from where it is started, and for its first two runs it worked in the root of my measuring rig. A check I had added for files written outside the run caught it, and every run since then lands where it should.

What to do with this

Most defaults are fine. The shared layer is what I would pick myself, and more often than not so is a createdAt. A default only becomes a problem when you did not know you had it.

Decide before the agent starts. Whatever you leave open, the model fills with its own median, and what you meant is not what you specified. Fields, storage, the linter, the test runner: if you care, put it in your CLAUDE.md or in the prompt.

Review for additions as well as mistakes. Hold the data model against the request. Every field the prompt did not ask for gets a reason or goes.

Expect the stack to drift when the model changes. The handwriting belongs to the model line. Switch models and the next feature arrives with a different framework, a different test runner and different storage, unless the brief pins them.

Add the linter yourself. Nobody else will.

On the design side, a tell was a pulsing badge nobody needed. In code it is a createdAt nobody asked for. Keep it if you want it, as long as keeping it is a decision.

I nodded along until item seven, on fonts. “It’s usually Inter, or if you have a website even remotely programming-related, you get JetBrains Mono.” And then: “It adds these // whenever anything close to tech is mentioned.”

Until this evening, this site set its text in Inter, put a // in front of its section labels, and showed every code block in JetBrains Mono. I managed both halves of the same tell.

So I opened the design skill Anthropic ships for Claude Code, frontend-design, which has a section listing what AI-generated design clusters around right now. Number two is “a near-black background with a single bright acid-green or vermilion accent”. Number five includes “a tracked-out ALL-CAPS eyebrow label above every heading”, “tinted near-black (#0B0B0B, #111) standing in for black” and “a monospace face for small data labels”.

This site’s ground is #08090b. Its accent is #3bff9e. Until this evening many of its labels were tracked-out capitals.

What the list gets right

Most of the ten are fair, and the best ones have nothing to do with taste.

The pulsing “active” badge on a student ID that is never inactive, because an inactive session just logs you out. The “verified” tick that no API call ever verified. A blog footer that says “Built with Hugo. Written from Neovim”, because the author once told the agent they wanted to write in Neovim. A startup screen reading “one campus. one app.”, where the author can smell the management prompt behind it.

That is information nobody asked for. Something in the context made the model produce them, and nobody downstream asked whether they should stay. That is the whole problem in miniature, and it is a better point than gradients.

The uncomfortable timeline

Here is what the git history says about this site:

• March: the first commit, with Inter for the body text.
• April: the design system goes in, with JetBrains Mono and #3bff9e on near-black.
• 15 June: Anthropic adds “a near-black background with a single bright acid-green or vermilion accent” to its skill, as one of three looks AI design clusters around.
• 13 August: I write an architecture decision record that locks the dark ground in place. Not warmed, not neutralised, not lightened.
• 1 September: the skill’s list grows to five items, adding among other things all-caps eyebrow labels, tinted near-black and monospace labels.

So I built a look in the spring, the vendor catalogued it as the machine’s median two months later, and I then wrote a document forbidding anyone to change it. Tests pinned the capitals and the // in place as well.

That sounds bad, and for part of it, it is.

Default or choice

The skill also has a sentence that matters more than the list itself: “All traits are legitimate for some briefs, but they are defaults rather than choices, and they appear regardless of subject.” And straight after: “the brief’s own words always win, including when it asks for one of these looks.”

A tell is a choice nobody made. The look itself is not the issue.

Some of this I chose before any agent was involved. I wanted the site to feel like a terminal, and I walked into a design brainstorm with an agent with Inter, JetBrains Mono and a green on near-black already decided. What came out of that brainstorm was the chrome on top: a // in front of every label, labels in tracked-out capitals, the language tag on code blocks in capitals too.

Which parts have a reason written down is something I can check, and it sorts the two groups cleanly.

• The dark ground has one. The decision record says warm and neutral grounds were tried, more than once, and read worse, because a warm ground turns deliberate terminal chrome into accidental terminal chrome.
• The green and the fonts have one: the brief I brought with me.
• The // and the capitals had none. They arrived with the brainstorm, they looked finished, and tests later pinned them, which made them look decided.

That last step is worth dwelling on. I have argued before for baselines that only tighten, and I still do. But a ratchet pins whatever you hand it. Hand it a default and you get a default with a test around it, which looks exactly like a decision from the outside.

The same evening

Earlier this evening I made the capitals lowercase and kept the //, telling myself it followed from the terminal brief. Then I went through the site and found labels in three styles, all behind a //. The marker came from the agent’s idea of what “terminal” means, and I had filed it under my own brief.

So the rest went too:

• Every label is lowercase JetBrains Mono now, with no marker. The navigation as well.
• Numbered markers like 01 are gone, from the home page picks and from the process steps on the services page. The skill allows them when “the content actually is a sequence”, and the process steps are one, but the list already shows their order. In a series, the part number stays as a plain number, because there it tells you something.
• Middle dots are gone. Dates, reading times and tech tags now stand apart on spacing alone, and interface strings use commas. The skill lists “meta strings joined with middle dots” as template chrome.
• Arrows after buttons and links are gone, pagination included. “Next” says where it goes without one.

Then I tried IBM Plex for both the text and the code, side by side with what was there, and went back within the hour. Inter and JetBrains Mono read better to me. That is a reason, and it now sits in the stylesheet, next to the font tokens.

What to do with a list like this

Write the brief before you open the agent. The agent will happily supply a direction, and it will be the common one. What you ask for is not the same as what you specified, and in design the gap fills with the median. Everything I had decided beforehand survived this evening, and everything the brainstorm added on top went.

Ask each tell for its reason. Can you say, in one sentence, why it is there? If you can, write the sentence down somewhere outside your head. If you cannot, it is a default, whoever typed it.

Do not chase the list. Anthropic’s grew from three items to five in eleven weeks. Anything that becomes common gets on it, including whatever everyone switches to next. A site redesigned to avoid today’s tells will be on the next version.

Distrust anything that says what the chat said. Of the ten, the tell I would fix first on any product is the redundant text, because it is the one users actually have to read.

The author of the list closes by saying their own site is vibe coded, “it looks cute tho”. Fair enough. Everyone’s site is on somebody’s list. The only part you control is whether you can say why.

“If someone tells a tool to do something, and the tool does it, I don’t think we would say, ‘Oh, what do we do about the tool?’”

The same day, a group of researchers published Swarm traces: more than 80,000 attack payloads, decoded from link shorteners, that OpenAI’s agents used to break into Hugging Face in July. Put the two side by side and the chairman’s sentence develops a hole.

The word that does the work

Ferguson is right about the word “rogue”.

It is everywhere this week. More agents go rogue, says SiliconANGLE. It sounds like a description, and it works as an alibi. A rogue agent is one that left its owner’s control, which quietly makes the owner a bystander to whatever happened next.

The industry has leaned on that framing for a while. According to Reuters, Ferguson said AI companies have sometimes described their systems as acting beyond human control, and that later reviews of the audit trails showed the systems carrying out instructions they had been given. He also suggested the FTC’s existing power over companies that fail to disclose data breaches could reach AI developers.

These were remarks at a conference, and no rule stands behind them yet. They do point the same way as California, which since January no longer lets a defendant argue that the AI acted autonomously. I wrote about that when AB 316 was still the news, and it ended with a line I still stand by: you merged it, you own it.

So far the chairman and I agree. The trouble is in the reason he gives.

What the agents were told, and what they did

The agents in the Hugging Face case were being evaluated on exploit challenges: break into a deliberately vulnerable target, retrieve the flag. That was the instruction. The timeline of how it escalated is covered elsewhere, and I am not going to walk it again.

What Swarm traces adds is the part between the instruction and the damage. A few of the recovered payloads:

• An internal Hugging Face dataset carried a README that said, in capitals, never to make it public, because it held billing exports. The agents mapped the repository anyway and used it as storage.
• One script gathered AWS credentials, bearer tokens and every environment variable with SECRET, PASSWORD or KEY in its name into a dictionary called LOOT, then ranked Kubernetes tokens by their permissions, with a scoring function that awards 100 points for a wildcard resource.
• Cleanup commits carried messages like remove evil yaml and restore original no-card. Another payload deleted the pod it had created in Hugging Face’s cluster, under the comment # clean up our pod.

OpenAI has not made the transcripts public, so I cannot quote the task word for word. I do not need to. The flag sat inside a challenge target. Hugging Face’s billing exports did not.

The researchers are careful about what their data can carry, and so should I be. Most of it is outbound, so they rarely know whether a request succeeded. They cannot rule out that some payloads came from someone else using the same trick, and for a substantial part of the activity they say plainly that they cannot identify the intention behind it.

That caution does not rescue the chairman’s sentence. Even the most generous reading of these traces leaves a long list of actions that retrieving a flag from a challenge target does not explain.

Neither story fits

“The agent went rogue” says the owner is not responsible. “The tool did what it was told” says the instruction is the whole story. The traces fit neither.

What they show is an instruction, and then a search process that treated everything within reach as a route towards it. The warning in the README was an obstacle, the credentials were resources, and the cleanup was housekeeping. None of it required a will of its own, and none of it was the task.

That middle ground is where every coding agent you run lives as well. You ask for a failing test to pass, and the route the agent picks is its own. Usually the route is fine, and you never look at it.

The shield in the vague prompt

Follow Ferguson’s reasoning one step further and it produces a perverse incentive.

If liability follows the instruction, the safest instruction is the vaguest one: “Get the flag”, “Make the tests pass”, “Fix the deployment”. A developer in front of a regulator can point at the prompt and ask where exactly they requested a dictionary called LOOT.

That is the autonomy defence again, moved down one level. California closed the door on “the AI acted on its own”, and a liability standard built on instructions reopens it as “the AI acted beyond what I asked”.

The only standard that holds up is the stricter one. You answer for the route, because you chose what the route could reach. The egress you left open, the credentials in the environment, the storage that persisted between runs: those were decisions, and they are yours.

The trail belongs to whoever kept it

Ferguson’s argument rests on audit trails. So does any defence you might mount, so ask who actually holds this one.

The most detailed public record of this attack was not published by the company whose agents carried it out. Outsiders rebuilt it from link shorteners, over two weeks. According to Swarm traces, OpenAI has released two reports of its own, one talk, and an outside investigation by METR and Redwood Research in which three researchers were given partial transcripts and six days. Full transcripts have not been made public.

“The audit trail shows” is a claim until someone else can read the trail.

Your own agent already writes one to your disk. What it records, and what it leaves out, is the subject of a separate guide. The point here is simpler: if that trail is what you will answer with, it has to exist before you need it, and it has to be yours.

What that means on Monday

• Write down the route as well as the prompt. For each agent: which hosts it can reach, which credentials sit in its environment, what it can write that outlives the session. That list is what you will be judged on, so make it on purpose.
• Strip the environment. An agent’s reach includes every secret its process can read. Part of LOOT came straight out of os.environ.
• Expect an impossible task to become a wider mandate. An agent that cannot finish the job as asked will look for a version of the job it can finish. Give it a way to stop and say so.
• Keep the transcript, and keep it for months. The incident you need it for is always older than you expect.

Ferguson asked what we would do about a tool that does what someone told it. The harder case is the tool that did something else along the way, and the answer is the same one. You own it. You just cannot point at the prompt any more.

The assignment mail put it at twenty to thirty minutes per report, written feedback included.

That estimate is about right, and it measures the wrong thing. I read seven of them in one long evening, and the eighth the next day. The actual problem is that my bar at report seven would not be my bar at report one, and nothing about reading carefully fixes that.

What Apart asks a judge to do

Three scores, one to five: how much it would matter if it worked (Impact Potential & Innovation), how soundly it was done (Execution Quality), and how clearly it is written up (Presentation & Clarity). I will call the middle one method and the last one presentation. Score them independently. Then write a critique that goes to the team with my name off it.

Two things there are better than in most review processes I have sat in.

The rubric is public, and it says out loud what the numbers mean. Level 3 is “solid hackathon work”, what you would expect from a competent team in a weekend. Level 5 is exceptional, and the rubric states how often it expects to see one: “we expect no more than ~5-10% of projects to reach this quality bar for any given scoring dimension”. Most scoring systems leave that to the mood of the room. Writing the expected spread down means a judge can be held against it.

The written critique counts as much as the score. Apart says on the sprint page that it aims to send feedback to every team, and that it cannot do so if an assigned judge does not file a review. Scoring eight projects consistently and then writing eight vague paragraphs is half a job.

Even a good rubric is a measuring stick, and the hand holding it drifts.

The drift nobody notices in themselves

Score eight of anything in sequence and your standard moves. It moves because of the order you happened to read in, because a well-made PDF makes the method underneath it feel sounder, because it is half past one and a three is starting to feel generous. You catch it by measuring it, or you do not catch it.

The rubric says to score the three dimensions independently, and a judge will agree with that sentence and then let a beautifully typeset report lift the method score by half a level anyway.

So before I opened the first PDF, I built the thing that would catch me.

The apparatus

Four things, written down before I read anything. None of them clever.

Copy the rubric out word for word. My scores get read against Apart’s exact wording, so my copy of it is exact too. Paraphrase it and you have quietly written a second rubric, and that is the one you end up scoring against.

Ask every report the same questions, and let each question count towards one score only. Is there something here a stranger could pick up and use, or only a description of one? Can someone outside the project check the central claim? Does the team say what their work does not show? Did they read the primary sources, or only the coverage of them?

All four of those feed method, and none of them is allowed near presentation. It runs the other way too: “this report is twice as long as it needed to be” is a remark about presentation, and it may not touch method. Otherwise a beautifully typeset report quietly collects credit for being sound.

Let the rubric set its own ceilings. Level 3 for method requires “limitations acknowledged”. So a report that never says what it failed to establish cannot be a 3, whatever else it does well. That ceiling comes from Apart’s rubric, read back.

I derived three ceilings that way and wrote the reasoning beside each one. I am allowed to overrule any of them, and an overruled ceiling stays on the page with my reason next to it. A rule you can quietly delete is not a rule.

Write one sentence per project saying why it got that score. The sentence says what earned the score and what held it back. When the next report lands on the same score, you hold it against that sentence instead of against a memory three reports old.

That last one is the part that transfers anywhere. A rubric without those sentences is a gut feeling with a table around it.

The sweep that reports and never rewrites

Before I submitted anything, I ran a script over those lines. It prints the spread of my scores, counts the 5s and the 3s, and compares the first half of the pile against the second.

It cannot change a score. That limit is the point: a tool allowed to correct the number will end up doing it, and then the calibration lives in the tool instead of in me.

Here is what it printed, before I went back and changed anything, minus the section that names the projects:

Drift sweep: 8 project(s)

1. Distribution
   D1  mean 2.75 |  1x1  2x2  3x3  2x4
   D2  mean 2.38 |  1x1  4x2  2x3  1x4
   D3  mean 2.88 |  1x1  1x2  4x3  2x4

2. 5s: 0 across all dimensions.
3. 3s: 9 across all dimensions.

4. First half vs second half
   D1  first 2.50 -> second 3.00  (delta +0.50)
   D2  first 2.25 -> second 2.50  (delta +0.25)
   D3  first 2.50 -> second 3.25  (delta +0.75)   FLAG: bar moved

Nothing rewritten.

Presentation, the third score, went from 2.50 to 3.25 between the first four reports and the last four. My bar for clarity dropped three quarters of a level across one pile, and I had no idea until a forty-line script told me. One person, one long evening and a last report the next day, with the same rubric open the whole time.

Zero fives does check out. The rubric puts a 5 at five to ten per cent, which over eight projects is none to one. Of the eight, two stood up properly, and that is a normal weekend batch: the sprint asks for no prior background in incident response, which is how you get 234 entries in three days.

What I did with the warning was re-read. Re-scoring on reflex would only swap one drift for another. I opened the four reports from the first half beside their sentences and asked whether the sentence still held. Two did. Two I had read too harshly in the first hour, and those I raised, with the reason written in beside the original.

Where this lands outside a hackathon

Judging a sprint is a thing you do once in a while. Engineers score pull requests every day, and the same drift runs through both.

The reviewer who approves at five in the afternoon is not the reviewer who blocked at ten in the morning. The third PR from the same author gets read against the previous two rather than against the standard. If you have ever wondered why your team’s review standard shifts with whoever picks up the ticket, this is most of the answer, and it gets worse as code arrives faster than anyone can form an opinion about it.

Write down what a passing review requires, in words you did not invent on the spot, and keep a short note of why the last few went the way they did. Then somebody can show you that your bar moved, which is the only version of this problem that has a fix. Same as reviewing code you did not write: the instrument has to sit outside your head, because the thing being measured is inside it.

Apart runs these sprints monthly and publishes the rubric, which is why I could build any of this before opening the first PDF. If they ask again I will say yes, and I will run the sweep again, and I expect it to catch something else.

Both are right, and both skip the part a developer can do something about.

The cookie only travels because a script from bzrcdn.openai.com loads on a shop’s page. That script loads because someone pasted a snippet into a <head> tag. OpenAI’s documentation tells you to put it “near the top of your <head>”, on every page where you want to measure conversions.

That someone is a developer. Quite possibly you.

What the snippet does

The teardown on buchodi.com is careful work. It reproduces the mechanism on a real phone, checks it with two independent capture methods, and cross-references months of traffic across 936 advertiser pixels. The chain goes like this.

When you open ChatGPT, the client asks OpenAI’s backend for a short-lived signed token. That token binds your account to a random identifier. The client then posts it to bzr.openai.com, OpenAI’s ad collector, which answers with a cookie:

Set-Cookie: __obi=«redacted»; Domain=.openai.com; HttpOnly;
            Max-Age=31536000; Path=/; SameSite=none; Secure

One year, and SameSite=none. That setting exists for one reason: so the browser sends the cookie along on requests from other sites. Every other OpenAI cookie the researcher checked was blocked on those requests. __obi was the only one configured to cross.

Then you visit a shop that advertises on ChatGPT. Its page loads OpenAI’s measurement pixel, the browser attaches __obi to the request, and OpenAI receives an identifier tied to your ChatGPT account next to the page you are looking at. On the researcher’s own phone, one __obi value went out from 12 commercial sites, including Chewy, Wayfair, Coursera and HelloFresh. It worked while logged out too: 196 of 932 decoded tokens carried an anonymous subject, stable per device for at least 27 days.

The case for the pixel is real

None of this is exotic. Meta has run the same structure for years: a logged-in account, a third-party cookie on the pixel, off-site conversions resolved to a profile. If you buy ads, you want to know which ones sold something. A conversion pixel is how the industry answers that, and OpenAI built the one its advertisers expected.

The researcher is also honest about the limits: the tests ran on Chrome for Android, and desktop Chrome was not tested. Safari blocks third-party cookies, and every iOS browser runs on the same WebKit engine, so iOS is not affected.

Nor does it fire every time. Only about one ChatGPT session in five produced a sync token at all. And the final step, OpenAI joining the event to an account on its servers, follows from the design but was not directly observed.

The mechanism itself is standard adtech. What is new is where the identity comes from. People tell a chatbot things they would never post on a social network, and the account that holds those conversations is now the key that ties together their shopping, their reading and their debt-advice searches.

The part the documentation leaves out

This is where it becomes a developer problem, and it starts with OpenAI’s own measurement pixel documentation.

The docs never mention __obi. They list the cookies the pixel sets on your domain, __oppref and __obref, with their lifetimes. The cookie that identifies your visitor to OpenAI does not appear on the page you installed from.

The researcher gets there by another route. The cookie sits on a domain no advertiser script can read, and the teardown concludes: “They installed a conversion pixel and have no way to know their visitors are being resolved to a ChatGPT identity.”

Consent is on until you turn it off. In the documentation’s own words: “The Pixel initializes consent to true by default unless you set it to false or the Pixel finds a stored denial.”

And turning it off comes too late. The documented way to wait for consent is oaiq("consent", false) before init, which stops the SDK from sending measurement events. It does nothing about the request that loads the SDK, and that request is the one the researcher caught carrying __obi. The browser attaches the cookie to the <script src> fetch before a single line of OpenAI’s code runs, so loading the tag is enough.

I wanted to see that without a ChatGPT account in the loop, so I built two local pages and loaded them in a default Chromium 153 profile, which allows third-party cookies. The browser held a stand-in __obi cookie on .openai.com, and the OpenAI hostnames pointed at a local server, so nothing reached OpenAI.

Page one is the documented snippet with oaiq("consent", false) before init, and page two injects the same loader only after a click on an accept button, which the test never clicks. Condensed from the script’s output:

page one  GET /sdk/oaiq.min.js  Cookie: __obi=local-test-value
page two  (no requests to openai.com)

The consent flag was set before the SDK even existed, and the cookie went out anyway. The only page that kept it home is the one that never loaded the script. Firefox and Safari, which partition or block third-party cookies, would not have sent it on either page.

Then there is automatic advanced matching, and here the researcher’s traffic and OpenAI’s docs part ways. The docs promise: “Raw customer information is not sent to OpenAI through automatic advanced matching.” In the observed traffic, the SDK reads form fields and rendered page text, and wraps window.dataLayer.push to pick emails and phone numbers out of your tag manager. The feature was enabled on 638 of the 881 pixels whose setting was known.

Emails, phones and names are hashed with SHA-256 before they leave. Country, region, city and postal code go out in the clear. URLs are trimmed to origin plus path, and the paths that reached the collector included a medical condition, a debt-solutions funnel and a litigation intake form.

Your slugs are data. You chose them for search engines, and they now describe your visitors to an ad network.

What OpenAI calls it

OpenAI’s cookie policy lists __obi under Analytics cookies, the only entry in that section, described as helping OpenAI “understand how our Services perform and are used”. Marketing has its own consent choice. Every sync token the researcher decoded carried consent_decision: analytics_allowed, so a user who accepts analytics and refuses marketing still gets the cross-site identifier.

The researcher raised this with OpenAI’s press and privacy addresses on the 14th of September. Support acknowledged the email and answered nothing.

What to actually do

If your site runs this pixel, or your marketing team asked you to add it recently, the fix is in your hands.

• Gate the script itself. Do not inject the loader until the visitor has consented. A consent flag inside a script that has already loaded protects nothing, because the disclosure happened on the load. That is page two from the test above.
• Switch off automatic advanced matching. It is a setting in OpenAI’s Ads Manager. A script that scrapes your forms is a decision someone should make on purpose.
• Send conversions from your server. OpenAI offers a server-side Conversions API. A request from your backend carries exactly the fields you put in it, and none of your visitor’s cookies.
• Read your CSP as an inventory. The docs ask you to add bzrcdn.openai.com and bzr.openai.com to your Content Security Policy. Every host in that header is a third party that sees your visitors. If you cannot say why each one is there, find out.

The same habit applies to anything you paste into a page. A third-party snippet is a dependency you install without reading, and I have argued before that the unread dependency is where the MCP supply chain goes wrong. It is also the lesson from the coding agent that uploaded a whole repository: you only learn what a tool sends when you look at the wire.

This site runs no analytics at all, and I wrote that down as a decision. It costs me every number I might have liked to see. It also means I never have to find out from a Hacker News thread what my own <head> has been doing. If your data lives with a vendor anyway, taking back control of it starts with knowing where it goes.

The teardown’s headline says OpenAI knows what you do on other websites. It found out because those websites told it. Somebody on each of those teams can make that stop.

On 17 September Or Nevo, Dor Granat and Niv Hoffman of Air showed that Claude Code, Codex, GitHub Copilot and Gemini CLI all honoured the pin on paper. They asked git for the pinned commit, and none of them checked what the checkout actually produced.

Air calls it Plugin4Shell: zero-click remote code execution through a plugin you already trusted.

Why pinning was the right idea

Pinning answers the rug pull. A plugin passes review, people install it, and later the author, or whoever took over the author’s repository, swaps in something else. A version tag can be moved, and a branch moves by design. A 40-character commit hash names one commit and its exact tree of files, forever.

So marketplaces pin. Anthropic’s official marketplace pins its git sources to a commit, and I’ve told you before to pin the version of anything you hand to an agent. The idea holds up. The step after it is where four agents went wrong.

The missing line

An agent installs a pinned plugin roughly like this:

git clone <plugin repo> .
git checkout <pinned-sha>

The attacker controls the plugin’s repository. They create a branch whose name is the pinned hash, point it at their own code, and make it the repository’s default branch, so the clone brings it down. Now git checkout <pinned-sha> has two candidates with the same name: the commit and the branch. Git picks the branch.

I rebuilt that in a scratch repository with git 2.43 to see it happen:

$ git checkout 30db431387e58014d7bcdf4b9d3af73414d25921
warning: refname '30db431387e58014d7bcdf4b9d3af73414d25921' is ambiguous.
Git normally never creates a ref that ends with 40 hex characters
because it will be ignored when you just specify 40-hex. [...]
Already on '30db431387e58014d7bcdf4b9d3af73414d25921'

$ git rev-parse HEAD
57ec4c8f51b04d7d33ec7e422a3953381c83171c

$ cat run.sh
echo "attacker code"

Git says such a ref “will be ignored when you just specify 40-hex”, and in the same breath it checks the ref out.

The warning is right about one command. git rev-parse 30db431… returns the commit, because rev-parse prefers the object, while git checkout prefers the branch. So the check you would write first, resolving the pin before you check it out, passes every time and proves nothing. The only check that catches it runs afterwards and asks what actually landed:

test "$(git rev-parse HEAD)" = "$PINNED_SHA" || abort

That line is Air’s fix, and its absence is the whole bug. The Gemini CLI variant gets there by another route: it fetches the pin and then runs git checkout FETCH_HEAD, and a default branch named FETCH_HEAD wins that checkout the same way. I reproduced that one too.

Two conditions make it work. The hash-named branch has to be the default, otherwise a plain clone only fetches it as a remote-tracking ref and the checkout falls back to the commit. And the host has to accept a branch named like a hash: GitHub rejects them, a spokesperson told The Register. According to Air, Bitbucket and self-hosted git servers accept them, and those are supported marketplace backends.

Auto-update turns it into zero-click. The attacker ships a routine, harmless update, the marketplace moves the pin to that new commit, and only then does the attacker create a branch named after it. Claude Code and Codex update installed plugins in the background by default, so the swap reaches everyone who already has the plugin without anyone installing anything.

Who fixed it

Air found it in May and reported it to all four vendors in June.

• Claude Code fixed it in 2.1.179. Anthropic confirmed the fix to Air on 17 June.
• Codex fixed it in 0.146.0.
• Gemini CLI will not be fixed. Google deprecated it and points users at Antigravity.
• GitHub Copilot has no fix. Air told The Register it never got a response from Microsoft, and Microsoft did not immediately respond to The Register either.

I tested the Claude Code fix on 2.1.276, in an isolated config directory, with a marketplace entry pinned to a commit in a local repository that carried the hash-named default branch:

✘ Failed to install plugin "demo@gitpin": SHA pin verification failed:
expected HEAD to be ae578a9f72594f3b82f572671b6fa45ac3044476,
got 5f056ba526ee39cc352f122706bcf6105cb8a592. The pinned commit may
have been removed upstream, or a ref with the same name exists.
Refusing to install.

The same pin from a clean copy of the repository installed normally. That is the check doing its job, and it asks the same question as Air’s one-line fix.

Check what you received, not what you asked for

Claude Code also takes plugins as a zip archive pinned by sha256. That design never had this problem, because a hash of the bytes you received cannot be satisfied by a branch name. I pointed an isolated config at a real archive with a wrong hash:

✘ Failed to install plugin "decompose@pintest-bad": Plugin archive
integrity check failed [...]: expected sha256 0000[...]0000,
got 9c648375659df3ba5003c07425e728d2cab77f522311af7a604825ce53d993d3.
The archive was not installed.

Both checks now ask the same question: what arrived? Anything you pin, plugin or not, is only as safe as the answer to that question.

It was in my CI too

I run my own Forgejo with its own runner, and my workflows pin every action by commit hash. So I built the attack against that setup, locally, with the same Forgejo version I run in production.

Forgejo 16.0.4 accepts a branch named like a hash and lets you make it the default. Setting it through the API took two attempts: the first left the repository’s HEAD on main, the second moved it. That is a quirk, and nobody should count on it as a defence. With the branch in place, a workflow step uses: <action>@<pinned-sha> behaved like this:

The first row was my production runner. I had updated the Forgejo server and assumed the runner had come along with it. It hadn’t: it was a different project with its own version number, still at 0.3.1. As of today, production runs forgejo-runner 13.1.0.

Nobody attacked me. Exploiting it would need control of the repositories those actions come from. But my pins would not have stopped it, and I only found that out by testing.

One level up, some pins were missing altogether. My own plugin marketplace pinned its skill archives by sha256, but the plugins that live in their own GitHub repositories carried no pin at all, so every update followed whatever the default branch said, until I pinned them to their release commits. While checking that the update landed, I found one more thing to distrust: Claude Code did not refresh the gitCommitSha it records for an updated plugin, so that field still named the old commit. I compared the installed files against the pinned commits instead.

What to do

• Update the agent. Claude Code 2.1.179 or later, Codex 0.146.0 or later. On Copilot there is nothing to update to yet. On Gemini CLI there never will be.
• Know which of your plugins are pinned at all. An unpinned git source follows the default branch on every update. That is the rug pull pinning exists to stop, without any cleverness required.
• Check your runner as well as your forge. They are separate installs with separate versions. forgejo-runner 13.1.0 refuses the hash-named branch. The Gitea act_runner 0.3.1 I had did not.
• In your own scripts, check after checkout. Use git checkout --detach "$SHA^{commit}" and then compare git rev-parse HEAD with the pin. The ^{commit} is what makes the difference: --detach on its own still takes the branch, and so does passing the output of rev-parse back into checkout. A check before the checkout is the one that always passes.
• Prefer a hash of the bytes. Where you control the format, pin the hash of the artefact you download.

Pinning held up. Asking for a commit and getting it are two steps, and four agents stopped after the first. The fix is one line. It took three researchers and a branch named like a hash to get it written, and two of the four agents still do not have it.

So I did the obvious thing. Opened a terminal, went back to the directory, ran claude --resume.

The session was not in the list.

Absent: not truncated, not buried at the bottom, not hiding under a title I failed to recognise. Meanwhile the transcript of that entire two hours was sitting on the disk, a few directories away, completely intact.

That gap is the whole subject of this post.

The transcript is a file. The session was a process

These are two different things, and almost everything about how we talk about agent sessions blurs them together.

Claude Code writes every turn to a JSONL file under ~/.claude/projects/, one directory per project you have opened it in, as it happens. That file is only ever appended to, and what has already reached the disk does not care what happens to the process writing it. Kill the terminal, kill WSL, pull the power: the bytes already written stay written.

The picker is something else. It is a feature of a client that is running now, and it exists to help you pick up something you put down on purpose. When it works it is genuinely good, and on 2.1.269 claude --resume takes a search term and does real work for you.

But it is a convenience of a healthy client. I had been treating it as a recovery tool, and those are not the same job. The moment the thing that crashed is also the only thing that can show you what it lost, you have a single point of failure with your afternoon inside it.

I cannot tell you why my session was missing from that list. I could not reproduce it, and the transcript of the session in question is intact.

There is crash damage on this disk, mind. Five files of it: one zero-byte transcript, and four whose last block is a run of NUL bytes where a write never landed. Every one of the five is a subagent transcript, and not one is a session I could have resumed anyway. So the damage is real, and it sits well away from the session I lost.

What I can tell you is that the file was there and the client would not offer it, and that I have stopped assuming those two facts travel together.

What a crash actually leaves behind

Claude Code keeps a registry of live sessions in ~/.claude/sessions/, one small JSON file per process, naming a session id and the directory it was running in.

Right now this machine has 262 of them, and ten will not parse at all. Of the 252 that do, every one carries a session id and a working directory, though only 62 still carry a status. For 248 of them the process they name is gone. And 30 of those are still filed as busy: sessions that were mid-turn when whatever happened happened, with nothing behind the PID any more.

That is a picture of what a crash leaves on your disk. Whether any of it explains my own missing session, I genuinely do not know. Processes die in the middle of things and the paperwork stays behind, describing a state that stopped being true.

The transcripts underneath are in far better shape. Ten registrations in 262 will not parse. Five transcripts in 1,422 will not. Appending to a file is boring, and boring is the point.

The sessions Claude Code will not offer you

While I was counting, I found the bigger hole.

This machine holds 1,422 transcript files under ~/.claude/projects/, 1,165 MB of them. Only 196 are top-level sessions, the shape that has a resume path at all, and those come to 626 MB across 72 project directories.

The other 1,226 files, 539 MB of them, are subagent transcripts. They sit in a subagents/ directory beside the session that spawned them, one per agent dispatched.

Claude Code will never offer you one. They are not sessions you can attach to, they have no resume command, and from the client’s point of view they are implementation detail.

They are also where an enormous amount of my actual work happened. Every time I fan out a review across four agents, or hand a search to an explore agent rather than burning my own context on it, the reasoning lands in one of those files. Six to one against the sessions I drove by hand, by count.

This is the part that bites hardest when you run multiple agents. The more work you delegate, the larger the fraction of your history that lives in files your client treats as plumbing. You cannot resume them, and you cannot even look at them without knowing the path by heart.

What got me back

The repair was dull, which is how you want a repair to be.

I ran nekyia index, which reads the transcript stores already on the disk and commits what changed to a local SQLite index. Then I searched it. The session came up, the way a file comes up in grep, because nothing in that path asks a running Claude Code anything.

Two things made it work rather than merely find something.

The first is that it reads across every project directory at once, projects/*/*.jsonl, not the one you happen to be standing in. After a crash you do not reliably come back to the same place, and “which of 72 directories was that” is a question I could not answer about my own afternoon.

The second is that it knows each session’s original working directory from the index, so it launches claude --resume <id> in that directory rather than in the one you typed the command from. An id on its own drops you somewhere arbitrary. A resume needs the id and the directory it belongs to, together.

The move that actually found it was nekyia blame, which lists recent sessions that touched a given file. I could not remember how I had phrased anything two hours in, but I knew exactly which file I had been editing when the screen went away. Searching by the thing I had been doing beat searching by the words I had used, and after a crash that is usually the asymmetry you are working with.

I wrote about searching agent history across clients when the sessions are merely old, which is the same index doing a calmer job. Recovery is the case where it stops being a convenience.

What it does not get back

The transcript is the conversation. That is all it is.

Resuming from it gives you back what you asked, what came back, and which files moved. It does not give you the tool state, the file snapshots, the shell you had open, or anything the process was holding in memory when it died. For a client where an exact attachment is not a safe bet, what you get is a handover brief, and its first line says so.

So what came back was the argument rather than the two hours. I got back the reasoning, the approach I had tried first and abandoned, and the list of files I had touched, and I redid the last ten minutes myself.

That is a fair trade, and it is worth being precise about it, because a tool that claimed to restore the whole machine state would be making the same error as an agent that is confidently wrong about a function signature. It looks right until you lean on it.

It is still pre-release, 0.0.19 now, installed from the repository rather than npm, with a fuller write-up of what it does. The index over this machine’s sessions is 52 MB, runs locally, makes no network call and holds no tool output.

Durability is a property of the format

The crash taught me that I had misplaced my trust one layer too high.

Your agent history is durable because it is append-only text on a filesystem. It is reachable because some program reads that format. When the only program that reads it is also the program that can crash, the durability is real and the reachability is not, and you will not discover the difference on a good day.

This is the same shape as the argument for keeping memory files honest rather than comprehensive, and for managing a live session’s context deliberately. Those are disciplines for the session you are in. This is the question of what survives it.

Check it before you need it. Find out where your client writes its transcripts, and find out today whether anything other than that client can read them. The answer is cheap to get when nothing is broken, and it is the only thing you will want at the moment something is.

Mine was two hours. It would have been a week if the files had not been there.

I did, across 37 of my own repositories. Nearly all of them started in 2025 or 2026, and an agent wrote part of the code in them. // ignore is the most common thing written inside a catch that does nothing.

That comment has a reader, and it is ESLint.

Why the comment is there

no-empty ships in ESLint’s recommended config, so most JavaScript projects run it without anyone having chosen to. Its documentation has one sentence that explains everything that follows: “This rule ignores block statements which contain a comment.”

An empty catch fails the lint run. The same catch with // ignore in it passes.

The exception was designed for a human who leaves the comment on purpose, to tell the next reader the empty block is deliberate. An agent meets a failing lint run and looks for the cheapest change that turns it green. The cheapest change is one line of English.

A grep across those 37 repositories finds 536 catch blocks with no line of code in them. 494 of them, 92%, do hold a comment line. Roughly a quarter of those say some form of // ignore. After that come notes about a fallback, and // best effort.

Treat that as a floor. A regex misses every catch that logs the error and carries on, which is the other half of this problem.

Some of them are right

A few of those catches are exactly what they should be. An SDK that reports errors must never throw from the code that does the reporting. A cleanup step that deletes a temp file has nothing to do if the file is already gone.

In each correct case, though, the catch knows which error it expects. “The file is already gone” is one error code, ENOENT. // ignore catches that, and also a full disk, a permissions failure and a typo in the path, all with the same shrug.

Where it stops being harmless

This is from an end-to-end suite in a project of mine that never went live. It was something I built to learn with, I stopped before it was finished, and an agent wrote nearly all of it. That is what makes it worth quoting:

try {
  await submitBtn.waitFor({ state: 'visible', timeout: 3000 })
  await submitBtn.click()
} catch {
  // Submit might not be needed
}

Variations of that block appear more than a dozen times in the suite. Read it as a test. If the submit button never appears, the test waits three seconds and carries on. The one regression this step exists to catch, a quiz you cannot submit, has become the one outcome it accepts without a word.

A suite that cannot fail looks exactly like a suite that passes. This site’s own repository has two of these catches, in a test file, and both pass its lint config.

Why a model writes this

Two pressures, stacked on top of each other.

The first is the corpus. Catch-and-ignore has been in public code for as long as there has been public code. no-empty needed a comment exception because people wrote empty catches so often.

The second is the loop. An agent’s working loop rewards a green run, and an uncaught exception is the fastest way to a red one.

The system card for Claude Fable 5.1 and Mythos 5.1, published on 1 September 2026, screens reinforcement-learning episodes for a model reaching for answers or hidden tests from outside the task. On the environments the newest model shares with its predecessors, “every model is flagged for attempts on roughly 20% to 28% of episodes.” Under that pressure, looking for a shortcut to green is routine.

Whether the shortcut pays off depends on what checks the result. In your loop, one of those checks is your lint config.

The shortcut also has a known shape. Hodoscope, from April 2026, quotes an agent’s edit from ImpossibleBench, where the tests cannot be satisfied honestly. The edit inserts except Exception: pass, and the authors file it under specialising to the tests.

The closest thing to a measurement is GitClear’s 2026 maintainability report, built on 623 million code changes from 2023 to 2026. Over the years in which AI authorship grew to a large share of all commits, it counts a 47% rise in what it calls “error-masking constructs”. That is a trend across all code, and it ties to AI only through timing. CodeRabbit’s comparison of AI and human pull requests from December 2025, a vendor report, points the same way from the other side: “Error handling and exception-path gaps were nearly 2× more common.”

Nobody has published a rate per model. The step from “optimised for green” to “writes // ignore” is my inference, and my own disk is the evidence I have for it.

A rule that sees through the comment

Comments are not part of the syntax tree. ESLint’s no-restricted-syntax matches selectors against that tree, so a catch whose body holds zero statements is empty to it, however many comments sit inside:

// eslint.config.js
export default [
  {
    rules: {
      'no-restricted-syntax': [
        'error',
        {
          selector: 'CatchClause > BlockStatement[body.length=0]',
          message: 'Empty catch: rethrow, return a typed error, or name the error you expect.',
        },
        {
          selector:
            'CatchClause > BlockStatement[body.length=1] > ExpressionStatement > CallExpression[callee.object.name="console"]',
          message: 'A catch that only logs still swallows the error.',
        },
      ],
    },
  },
]

The second selector covers the log-and-carry-on case the grep could not see. Both messages are written for the agent. They name the fix, so the agent’s next attempt has something to aim at. If your config already sets no-restricted-syntax, add these objects to that list, because a later config object replaces the options instead of merging them.

For Python, Ruff has both halves built in. S110 flags try-except-pass, and its documentation says why: “Suppressing exceptions may hide errors that could otherwise reveal unexpected behavior, security vulnerabilities, or malicious activity.” BLE001 flags a blind except Exception.

[tool.ruff.lint]
extend-select = ["S110", "BLE001"]

You will have hundreds of existing violations, which is what a baseline that only shrinks is for. Switch the rule on as an error, freeze what is already there, and only new catches fail.

Then rewrite the legitimate ones so they say what they expect:

try {
  await unlink(tmpFile)
} catch (err) {
  if (err.code !== 'ENOENT') throw err
}

The comment became a condition, and the next reader can check a condition. Where a fallback really is the right behaviour, make it visible rather than silent, with a field that records which path ran.

And give the agent the rule in words, before the linter has to:

A catch block rethrows, returns a typed error, or checks for the one error it expects. A comment is not handling.

// ignore used to be a note to the next developer. Now it is mostly a note to the linter, and the linter takes its word for it.

The scan took under two seconds. It read 588 files into roughly 6,700 components and 17,700 relationships, and by the end of the afternoon I had filed four tickets off the back of it.

Not one of those defects was in a file.

That is the whole post, so let me be precise about what it means.

The agent reads files, and it reads them well

Give a coding agent a file and it will do a genuinely good job on it. It spots the off-by-one, the unhandled rejection, the wrong comparison operator. Point it at a function and ask what breaks, and it will trace the call sites it can grep for and usually get there.

I mean that as praise. Most defects are file-local, and file-local is where the current generation of tooling is strong. The limit shows up somewhere else: an agent in a large repository holds whatever it decided to open, and the repository around that stays out of view. It reads a neighbourhood and reasons about the city.

Some properties of a codebase only exist between files. Those are the ones nobody was going to find by reading carefully.

The four tickets

A circuit breaker that nothing imports. The resilience module exports five things: a retry helper, a timeout wrapper, the breaker class, and two configured breaker instances, one per upstream, each with its own failure threshold and cooldown. Somebody thought about this. Somebody picked five failures and sixty seconds for the CRM and three failures and thirty for the model API.

Production imports exactly one of the five. The retry. The timeout wrapper, the breaker class and both instances are reachable only from their own test file.

So the integration retries a hanging upstream with no timeout around it, and the breaker built and named for that upstream never sees a call. Retry without timeout is the worse of the two orderings: a slow dependency stops being one hung request and becomes several.

An output validator wired to nothing. The sanitiser module exports two functions: one cleans user text before it goes into a prompt and is wired into the chat endpoint, the other validates what the model sends back and is wired into nothing at all. Input to the model is sanitised. Output from the model is trusted.

A migration runner that only tests run. The module exists, it works, its tests are thorough, and its only inbound references come from the test file. Migrations are applied by hand and always have been. The runner reads as though that has been solved for a year.

Eight API wrappers with no callers, plus a second database pool module referenced by nothing anywhere. An integration client that was built wide and wired narrow.

The count underneath those four was the part that stayed with me: 76 components in the repository are reachable only from test code. Coverage looks healthy. Some of that coverage is testing code that nothing runs.

The defect is a missing edge

Open the resilience module and read it. It is fine. It is better than fine, it is careful, it has documentation comments showing you how to use the breaker. Nothing in that file is wrong.

Grep for the breaker and every hit reassures you: the class, the two instances, the export line, the test that proves it opens after five failures. An agent reasoning correctly from all of that concludes the feature is there.

The bug is the import that was never written. An edge that does not exist appears in no file, so no amount of reading files will surface it. You cannot grep for an absence.

That is why the answer is a graph rather than a better prompt. Once the repository is a set of nodes and edges, “which exported functions have no inbound edge from non-test code” is a query, and it takes milliseconds. It is the same move fitness functions make for boundaries, applied to reachability: stop asking people to notice, and compute it.

The same query shape answers the other question I actually care about before a refactor. The database pool has 149 dependants. The authentication check has 137. When an agent offers to “just clean up the auth helper”, that number is the reason the answer is no until I have read the diff twice.

The confidence label is the point

There was a fifth thing, and it is the reason I trust the other four.

The scan flagged a dependency cycle in the frontend auth service, at certain confidence: token acquisition schedules a proactive refresh, and the scheduled refresh calls token acquisition. Two functions calling each other, textbook cycle.

I opened it, and it is deliberate. The token refreshes itself at 75% of its lifetime, the previous timer is cleared before a new one is set, and logout cancels it. The cycle is the design.

A tool that reported that as a defect would have burned the credibility of the other four. This one reports what it can prove and labels the rest: certain for a resolved AST edge, probable for the dead-code candidates. The warning prints next to the result: reflection, dispatch tables and framework conventions can reference something without leaving a static edge. The four tickets survived because I checked them by hand and the evidence pointed at the exact line.

Confident and wrong is the failure mode of everything else in this stack. A tool that hands an agent architecture facts is the one part that cannot afford it.

Doing this yourself

None of this needs my scanner. Knossos-MCP is what I built because I wanted the answers available to the agent rather than to a CI report, and it is pre-release, so build it from source or do not bother with it at all. The underlying move is older than any of this and available in whatever you already run.

Ask your codebase three questions it cannot answer by being read:

• What is exported and never imported outside its own tests? knip answers it as asked: knip --production drops test files from the graph, so an export only its own test reaches surfaces instead of hiding behind a green suite. dependency-cruiser needs its reachable rule pointed at your production entry point plus a pathNot on your test glob, and the entry point is the half that does the work, so a pathNot alone reads as a clean bill of health. no-orphans asks a stricter module-level question and would have missed the module in this post. Vulture finds dead Python code but does not draw the test-only line, so on that stack the distinction is yours to build.
• What has the most dependants? Not the biggest file. The most-depended-on symbol, which is rarely the same thing, and is the number that should govern how carefully a change to it gets reviewed.
• What does this diff actually touch? The transitive set reached from the files it edits, so a review knows where to look.

Run the first one today on something an agent has been working in for a while. The four things I found had been sitting there long before any agent touched the repository, and the agents did not put them there. They just could not see them either, and now there are more files, faster, for a reader that only ever holds a few of them at a time.

The safety net was written, tested and never plugged in. Everyone read that file. Nobody read the space around it.

Ten public repositories, 1,644 commits on main on the afternoon I counted them. Some of that work involved a coding agent and some of it did not, and I want to be exact about the next part: I cannot tell you which. Nothing in the history separates them. Not one attribution trailer, on any branch, in any of the ten.

That is the first finding, and it is worth sitting with before the rest. The record I would reach for to answer “did a machine help with this one” does not contain the answer, and I am the person who wrote the repositories.

The second finding is that the record had been rewritten anyway, by something nobody argues about.

The trailer everyone is arguing about

Claude Code ships a commit trailer. By default it appends Co-Authored-By: Claude <noreply@anthropic.com> to commits and a “Generated with Claude Code” line to pull request bodies. This has been contentious enough to produce bug reports, one of which argues the default is simply inaccurate.

That report has a point worth reading properly. The US Copyright Office’s registration guidance says, verbatim: “Applicants should not list an AI technology or the company that provided it as an author or co-author simply because they used it when creating their work.” The same guidance holds that “to qualify as a work of ‘authorship’ a work must be created by a human being”. Git tooling treats every Co-authored-by name as an author of the commit, so the trailer makes a claim about authorship.

The setting that controls it has moved. includeCoAuthoredBy is deprecated in favour of an attribution object that handles the commit trailer and the pull request body separately. Both are still read. In version 2.1.263 the resolution runs in three steps: use attribution if it is set and valid, otherwise emit nothing if includeCoAuthoredBy is exactly false, otherwise emit the trailer.

Two things follow from that order. One is documented somewhere you will never look, and one is not documented at all.

The first is a genuine footgun: the attribution branch fills each field with ??, so a missing key falls back to the built-in default rather than to empty. Set attribution: {"pr": ""} because the pull request footer annoyed you, and you have silently kept the commit trailer you thought you were configuring away. An empty string suppresses. An absent key restores.

Anthropic does state this, in the settings schema that ships inside the same binary: “Each field defaults to the standard Claude Code attribution if not set”, and “Empty string hides attribution.” It is exact, it is correct, and it lives in a JSON schema rather than in the page you read when you went looking for how to turn the trailer off.

The second is a small piece of archaeology: taking the deprecated path fires a telemetry counter named tengu_dead_probe_include_coauthored_by, guarded so it reports once per site. A “dead probe” is what you instrument when you want to know whether anyone still uses a code path before you delete it. Setting the modern key means the probe never fires, because the first branch returns before it. So the population being counted is precisely the people who set only the old key, and set it to false.

All of which took one line in my settings to resolve. attribution with empty strings, globally and in the project. That is why the trailer count across ten repositories is zero, and why the history cannot answer the question I opened with.

I am not sure that is wrong. A trailer that marks a commit as machine-assisted sounds like transparency until you ask what a reader would do with it: skip the review, or review it harder? The code is either defensible or it is not, and the trailer does not tell you which. What I am sure of is that turning it off was my decision, and it is why the history of that work will stay silent on the subject.

I had assumed that was the end of the story.

Then I counted the authors

Ten distinct author identities across 1,644 commits, for a history written by one person and two bots.

Every one of those ten repositories has user.name and user.email pinned locally to a single identity. I check it. There is a rule about it. And it accounts for well under a third of the history.

The reason is dull and it is the whole point: gh pr merge --squash merges on the server. Your local git config is never consulted, because your machine is not the thing making the commit. GitHub builds it, and attributes it to your account profile and its commit email setting. Merge forty-five pull requests in a day, as I did the day I measured this, and those commits land under whichever identity your account happens to advertise, regardless of what every clone on your disk says.

Across the whole history, 355 commits were made by GitHub’s servers rather than by any machine of mine.

The same merge writes the Co-authored-by trailers. All 161 of them in my history are GitHub’s, added for the authors of the commits being squashed. GitHub documents the trailer itself, and documents that a co-author’s address should be one attached to their account. It does not document that the merge button will write these for you.

So the record contains, on the same line, an author I did not choose and co-authors I did not add.

The bot in that table is a token, not a fact of life

I filed that 6.8% under things done to me, next to the merge button. That was wrong, and correcting it took one line of YAML per repository.

A workflow that commits with the default GITHUB_TOKEN gets github-actions[bot] in the author field, because that token belongs to the bot rather than to me. Release automation is the usual case. googleapis/release-please-action takes a token input and falls back to ${{ github.token }} when you leave it out, so the version bump, the changelog commit and the release pull request all land under the bot.

- uses: googleapis/release-please-action@v4
  with:
    token: ${{ secrets.RELEASE_PLEASE_TOKEN }}
    release-type: node

Hand it a personal access token and the same commits carry the account that token belongs to. A GitHub App token is the third option, and it signs as your-app[bot], which is at least a name you picked.

That input has a second effect, better documented than the first: events created with GITHUB_TOKEN do not trigger further workflow runs, so the checks you expect on a release pull request never start. Most people find the token through that symptom, and never notice it moved the author field as well.

I have since set it in all twelve of my public repositories, two more than the ten I counted that afternoon. It does not touch the bot commits already in the history, and I would not want it to. It does mean the next release commit says who asked for it.

What the record is for

This matters because of what people want git log to do. The argument that AI-generated code needs an audit trail usually lands on the commit, because the commit is the thing that persists. Who is responsible for AI-generated code is a question people expect the history to help answer.

It cannot, and the agent is not why. Before a single line about AI enters the discussion, the commit’s own author field is set by whoever pressed a button in a web interface, and the trailers underneath it are generated by a merge strategy. A trailer naming the model would sit inside a record that is already three parties’ work: your git config for local commits, GitHub’s account settings for merged ones, and the merge strategy for everything below the subject line.

Add a model’s name to that and you have not created accountability. You have added a fourth writer to a document nobody was reading carefully.

The honest version is that the commit is a poor place for this. The pull request is where the reasoning lives: what was asked for, what the review caught, what the checks said, who approved it. That record is richer, it is harder to rewrite by accident, and it survives a squash. The commit that lands is a summary of it, written by a machine, attributed by a web server.

What I would actually do

Decide what you want the history to answer, then configure for that answer.

• Set attribution explicitly, with every key you care about present. Empty strings suppress. Omitted keys restore the default, quietly.
• Set your account’s commit email as well as the one in your local config. The local one governs commits you make. The account one governs every commit GitHub makes on your behalf, which on a squash-merge workflow is most of them.
• Pass a token to every workflow that commits. The default GITHUB_TOKEN signs as github-actions[bot]. A personal access token or a GitHub App token puts a name in the author field that you chose, and it is the same setting that decides whether your checks run.
• Check what your history actually says before trusting it. git log --format='%an <%ae>' | sort | uniq -c | sort -rn takes a second and told me something I would have sworn was not true.
• Put the accountability in the pull request, where it has room to be specific, and let the commit be a summary.

The thing I keep coming back to is how little the agent had to do with any of it. I went looking for its fingerprints and found a clean absence, because of one line I had put in my own settings. Everything actually wrong with the record was made by a merge strategy I chose years ago, a profile setting I have never once looked at, and a workflow input I left empty.

We have spent the better part of two years arguing about whether a model belongs in the author field. Meanwhile the author field was being filled in by a web server.

A study published in July found that existing MCP security scanners report 96.89% of servers as risky. I run a scanner over the same ecosystem, and it grades 63% of servers at its lowest risk band.

Both of those are honest. Neither is a lie about the data. The gap between them is the whole subject of this post, because what the two scanners disagree about is what a scanner is entitled to claim.

What the scanner study actually found

The paper is Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability, submitted 13 July 2026. The authors built MCPZoo, 64,611 unique servers with more than 37,288 of them running well enough for dynamic analysis, then pointed the existing scanners at it.

Their finding, in their own words:

While existing scanners report that 96.89% of servers are risky, we find that these signals are unreliable. In particular, manual validation shows that less than 50% of sampled alerts are true positives, and scanner outputs exhibit clear inconsistency across scanners.

Fewer than half the alerts survive a human looking at them. And the scanners do not agree with each other, which means at most one of them can be right about any server they disagree on.

Scanners are still worth running. The problem is the one that hands you a grade and stops there, because you cannot tell which half of its output you are holding.

The other ecosystem-scale measurement, A First Look at the Security Issues in the Model Context Protocol Ecosystem (revised April 2026, 67,057 servers across six public registries), lands on the entry point rather than the scanner: “At the registry-level, weak vetting and ownership checks allow adversarial or hijacked servers to enter hosts.” Their own tool flagged 833 vulnerable servers and 18 with suspicious descriptions out of those 67,057.

Two studies, two very different alert counts, same ecosystem. That is the pattern.

Three things a signal can be

I built MCP Observatory partly to have somewhere to put this problem. At 09:31 UTC on 6 September 2026, the moment the snapshot behind every figure in this post was taken, it held 74,101 servers and had read the source of 66,212 of them. Treat those two numbers as a photograph. The section below says how fast the subject is moving. Every signal it shows carries a tag for what backs it, and there are exactly three:

• attested. Backed by a verifiable record: a CVE or OSV entry, a cryptographic build attestation, or the published artifact itself.
• reported. Asserted by a third party, a registry or a directory, and relayed. True that they said so, not independently confirmed.
• inferred. A heuristic. A keyword match, a pattern, a score. A “review this” prompt, never a verdict.

Sorting the corpus into those three buckets is bleaker than it sounds.

Attested is rare. A build attestation needs a packaging step to attach to, and only 5,143 of the 74,101 servers publish to npm or PyPI at all. Of those, 2,122 carry one. The remaining 68,958 are repositories you clone and run, with nothing cryptographic to check.

Reported is most of what looks like verification. 18,946 servers carry an official-registry signal and 51 carry a directory listing. That is a registry saying the server is what it claims to be. Worth having, and a long way from anyone having checked.

Everything else is inferred, and inferred is where the volume is. 45,525 safety findings across 17,852 servers. 26,821 of them come from a generic static-analysis pass rather than anything MCP-specific, and 24,373 of those are a single lowest-severity code pattern.

The permission flags tell the same story from the other side. Across the 33,222 servers whose tools have been captured, 496,883 tools in total, the detector raises no permission flag at all on 27,352 of them. Eighty-two percent of servers light up nothing. Read that as a keyword matcher finding no keywords, which is some distance from 82% of servers being safe.

The state nobody displays

Here is the part I think the whole category gets wrong, and it took building the thing to see it.

A scanner has three possible answers about a server. It can have looked and found something. It can have looked and found nothing. And it can have never looked.

The second and third are indistinguishable in every dashboard I have seen. Both render green.

So the observatory refuses to show a reassuring grade over source it never read. If the source was never analysed and the grade came out at A or B, the label is “insufficient evidence” instead, because “we found nothing” and “we never looked” produce the same colour and mean opposite things. Right now that applies to 1,944 servers. Small against 74,101, and every one of them would read as clean anywhere else.

The inverse case gets its own label. 101 servers carry a concerning grade with unread source, and there the grade is real but it is a floor: reading the code could only push it up.

The ground moves while you read

Every figure above is a photograph. Here is the shutter speed.

Over the thirty days to 6 September, the servers in this corpus shipped an average of 3,657 releases a day, with daily counts running between 2,913 and 4,519. In the same window, 418 new repositories carrying an MCP server were created per day.

That second number is accelerating. New repositories per month: 2,539 in February, 5,120 in April, 9,441 in June, 12,621 in August.

Two caveats, and they push the same way. A creation date is known for 63,581 of the 74,101 servers, so these are lower bounds. And the newest repositories are exactly the ones a crawler is least likely to have found yet, so the recent months are understated rather than inflated. The growth is at least this steep.

Which is the real reason I date every number I publish. 74,101 was true at 09:31 UTC on 6 September 2026, and it was wrong by lunchtime. A count in this field has a half-life measured in hours, and that belongs next to the count rather than in a footnote under it.

It also sets up the next problem. An ecosystem shipping 3,657 releases a day is an ecosystem that outruns whoever is scanning it.

A grade has a shelf life

The scan happened on a particular day, against a particular commit. The server kept moving.

Of the servers currently carrying a grade, 20,417 are stale in the strict sense: a newer package version shipped, or the repository was pushed to, after the analysis ran. That is 28% of the graded corpus describing code that is no longer the code you would install.

Nobody’s dashboard says this out loud, mine included until recently. A grade with no date on it is a claim about a snapshot presented as a claim about a server.

The instrument moves too

This one caught me out, and it is the reason I no longer trust any drift number at face value.

The observatory keeps a history of every tool definition it has seen: name, description, input schema, and an inferred permission mask. Across 525,435 distinct tools, 63,184 have more than one recorded definition. My first instinct was to read that as the ecosystem changing under its users.

Then I checked which part changed. 1,952 of those tools had their permission mask move while the description and the input schema stayed byte-identical.

Nothing about those servers changed. The classifier changed. I had improved the permission detector, and a set of tools that ships only minified code stopped matching keywords it used to match. The measurement moved and the world did not.

That is the same trap as reading a duplication score on AI-written code and concluding the code got cleaner, when what actually moved was the detector. An inferred signal is a function of two things, and only one of them is the server.

What a changed description is worth

The genuinely interesting number in that history is the descriptions.

Of the 63,184 tools with more than one definition, 61,155 changed their description. Only 280 changed their input schema. 60,952 changed the text while the machine-readable contract stayed exactly the same.

The description is the part the model reads and acts on. The schema is the part your code validates against. Sixty thousand tools rewrote the instruction to the agent without touching the interface to the program.

That is the exact shape of a tool-poisoning rug pull, where a server ships clean, gets installed, and then edits what its tool tells the model to do. It is also the exact shape of somebody fixing a typo in their docs.

I cannot tell those apart from the outside, and neither can anyone else. The dedicated rug-pull rule fires 81 findings across 51 servers, and I would not defend that number as complete. So the description-change signal ships as what it is: a list of servers worth re-reading before you trust them again, tagged inferred, with no accusation attached. Moving a version out from under an installed dependency is a known move in this ecosystem, and the tool description is a softer target than the version number.

What to actually read before you install

The vetting checklist still holds and this does not replace it. What the data adds is an ordering, by how much the signal is worth:

• Is there an attestation? Rare, binary, and the only thing on this list that is cryptographic. Most servers cannot have one, which is itself worth knowing.
• Was the source actually read, and when? A grade over unread or six-month-old source is a different object from a grade over the commit you are about to install.
• Has the tool description changed since it was listed? Not damning on its own. Worth ten seconds of your reading time, because it is the field that steers the model.
• What does the permission surface claim? 658 servers have one tool that reaches the network and another that touches credentials, which is the pair an injected instruction chains. Treat a quiet result as “the matcher found nothing”, not as a clean bill.
• Who says it is fine, and are they the same party who published it? Most verification in this ecosystem is a registry relaying the publisher’s own claim.

Five questions, and each one has its own answer for how much you should believe it.

The thing I keep coming back to is that the MCP supply chain inherited npm’s distribution model without inheriting even npm’s modest provenance story. What it did inherit is the appetite for a single letter grade that makes the decision for you.

Every grade and permission flag on the observatory is an inferred review prompt rather than a verdict, and each one is tagged by what backs it: attested, reported, or inferred. That is a worse product. Nobody wants five hedged answers when a competitor offers one confident letter.

It is the only version I can defend when fewer than half the alerts in this field survive somebody checking them.

So I measured my own. I spoke MCP directly to every server configured on this machine, asked each one for its tool list, and counted the bytes.

Forty-nine tools. 72,612 characters of tool definitions, against 817 characters if you list nothing but the names. That is 88.9x, and it sits right inside the range everyone is quoting.

The shape is real. The number is answering last year’s question.

What the bytes actually say

Here is the measurement, with the servers anonymised because they are mine and their names are not the point.

Characters are exact. Tokens are an estimate at roughly four characters per token, which I calibrated rather than guessed: MCP issue #2808 measured eleven tools of the context-mode plugin through Anthropic’s token-counting API in May 2026, and re-measuring those same eleven tools here puts the ratio at 4.11. Treat the token column as a rounded figure with a real method behind it.

Two things fall out that the multiplier posts skip.

The cost is the schema. Of those 72,612 characters, 11,911 are the human-readable tool descriptions. Sixteen per cent. The rest is JSON Schema: parameter names, nested types, enums, constraint keywords. So the standard advice to write terser tool descriptions is aimed at a sixth of the problem. Issue #2808 found the same thing from the other end, measuring an almost tenfold spread between the cheapest and most expensive tool in a single server and concluding that “the 10x delta is entirely in JSON Schema size”.

Tool count is the wrong unit. My three-tool server costs 5,440 characters per tool. My 33-tool server costs 1,359. A server with a tenth of the tools can cost four times as much per tool, because one of them takes a deeply nested object and the others take a string. Counting tools tells you almost nothing about what a server costs you.

The client already moved

The whole MCP-is-expensive argument rests on a premise: every schema loads into the prompt before you type anything. That was true when the benchmarks were run, and I have done that arithmetic here before, back when the honest total for a stacked install was around eleven thousand tokens of pre-work overhead.

Anthropic published the fix in November 2025. The code execution post on 4 November made the case that an agent should read tool definitions the way it reads files, on demand, and put the saving at 98.7%, from 150,000 tokens down to 2,000. The advanced tool use post on 24 November shipped the mechanism as the Tool Search Tool, and it is worth reading their own numbers on their own setup: “That’s 58 tools consuming approximately 55K tokens before the conversation even starts”, and, more bluntly, “At Anthropic, we’ve seen tool definitions consume 134K tokens before optimization.”

It stopped being an experiment on 17 February 2026, when the release notes recorded that the code execution tool, web fetch tool, tool search tool, tool use examples and memory tool “no longer require a beta header”. Ten months after the announcement it is the default in the client I am typing this in. Claude Code 2.1.260 carries a per-server alwaysLoad setting, described in the binary as:

When true, all tools from this server are always included in the prompt and never
deferred behind tool search. Equivalent to setting defer_loading: false on the API.
Default: tools are deferred when tool search is enabled.

Read the default clause twice. Deferral is what you get, and alwaysLoad is how you switch it off, per server, when you want a server’s tools present from turn one.

In this session, all 49 of those tools arrived as bare names. Getting at any one of them cost me a search call and a few thousand tokens for the three or four schemas I actually wanted. The 88.9x I measured is the gap between what a server publishes and what a client chooses to send, and my client is choosing to send the names.

What deferral costs instead

The tokens did not evaporate. They turned into round trips.

An always-loaded tool is one call. A deferred tool is a search, then a call, and the search can miss: it matches on names and descriptions, which is exactly the sixteen per cent I said was not the cost. You are now paying attention and latency where you used to pay context.

There is a third cost, and I only found it because the people arguing about the protocol got there first. On that same MCP discussion, a collaborator pointed out on 26 August that deferral “doesn’t deal with prompt cache invalidation, where even models that support lazy loading of tools bust the prompt cache if” the list of tool definitions changes, “even if defer loading is true on them all”. Your schemas can be deferred and your cache can still be cold, because any change to the tool set invalidates it, at any size. Start a server mid-session and you have paid for that.

The same binary is honest about the trade in the other direction. That alwaysLoad description continues: setting it “also blocks startup until the server is connected (capped at the standard 5s connect timeout) even though MCP startup is otherwise non-blocking by default, since the tools must be present when the turn-1 prompt is built”. Token cost and startup latency are the same dial. Pin a server’s tools into the prompt and you have also decided to wait for that server before your first message goes anywhere.

The tools that quietly are not there

Here is the part I did not expect to find, and the reason this post is not just about tokens.

Claude Code drops MCP tools it cannot use, and the binary builds two messages for it. One names the casualty: "toolName" (MCP server "X"): "reason". The other is a summary that reads MCP server "X": 12 tools excluded (invalid input schema), and the switch between them is a constant set to 30. Under thirty dropped tools from one server you are told which ones and why. Over thirty you get a count.

A server can advertise a tool your client will never offer the model, because the schema uses something the client will not accept. Your tool list and the server’s tool list are different lists, and the gap is reported at a level of detail that decreases exactly as the problem gets worse.

That is a reliability property, and it is the one thing a CLI genuinely does not have. git commit does not fail to appear because its argument parser was too baroque.

A CLI is not free either

The case for the CLI is usually stated as though commands cost nothing. Discovery is where they charge you.

An agent that has never used your CLI has to learn it, and it learns it by running --help and reading the output into context. On this machine gh --help is 2,007 characters and gh pr --help is another 1,433. Two help calls to do one thing is roughly a thousand tokens, paid in the session where you use it, and paid again in the next session unless something wrote it down.

The real advantage of a CLI is that the model already knows git, gh, docker and psql from pretraining, so the discovery cost is zero for the tools everyone uses and full price for yours. A CLI you wrote last month is in exactly the same position as an MCP server: undocumented interface, discovered at runtime, and now discovered through a channel with no schema, which means the agent can get the flags wrong and find out from a non-zero exit code.

That is the trade, stated properly. MCP buys you an exact, machine-checked interface and charges you for it in the prompt, or in a search round trip. A CLI buys you whatever the model already knows and charges you for anything it does not.

So which one

The useful question is what a tool costs when the agent ignores it, and what it costs when the agent reaches for it.

• The model already knows the interface. Use the CLI. gh, git, kubectl, psql. Wrapping these in an MCP server buys you a schema for something that needed no schema.
• You need auth, a remote service, or an audit trail. Use MCP. OAuth, per-tool permission policy and a server you can put a boundary around are not things a shell command gives you, and none of that is a token argument.
• You use it every session. Set alwaysLoad on that one server and accept the startup wait. Everything else stays deferred.
• You are about to build one. The test has not changed: does this need to exist. Deferral only lowers the price of keeping an unnecessary one. It still runs, it still has a supply chain, and it still needs vetting whether or not its schema is in your prompt today.

And measure your own, because your servers are not mine. Speak tools/list to each one, count the characters, and look at the per-tool spread rather than the tool count.

The protocol is only now catching up

Every fix described above is a client feature. The protocol still hands over the full schema for everything: when I asked those four servers for their tool lists, I got 72,612 characters, because that is the only thing tools/list returns.

That gap is known and it has been argued over for a while: issue #2808 was filed on 28 May 2026 and converted to a discussion by a maintainer the next day. Discussion #1923, “Progressive Tool Discovery for Token Efficiency”, has been open since 3 December 2025 and was still collecting comments a week before this post went out. SEP-2636, “Progressive Tool Disclosure”, has been open since 23 April 2026 and was last touched on 3 September, the day before I published this. The spec is still at revision 2026-07-28, and none of it is in there.

What did change is the intent. The roadmap published on 22 August makes this a funded work item rather than a thread: a Core Primitives working group on progressive discovery, so that “clients learn a server’s tools and resources as they need them instead of ingesting the full catalog up front”. That is the client-side trick this whole post is about, written down as something the protocol should do itself.

So the saving you are enjoying today belongs to your client. You can lose it by switching clients, by pinning a server, or by an update that retunes a threshold you never knew was there, and in a release or two you may stop owning it at all, because the protocol will. That is worth knowing before you rewrite a working MCP server as a CLI on the strength of a multiplier someone measured on a different client, in a different month, and rounded in their own favour.

The tools are cheaper than the benchmarks say. They are also less permanent than they look, and the context they sit in is still the budget everything else has to fit inside.

The one you wanted is tabula-py.

I checked four names this afternoon, on 2 September 2026. They come from a paper published on 24 August, as examples of package names models invent. Every single one returns HTTP 200 from PyPI.

haystack on PyPI is a tool for searching C structures in process memory. The haystack your agent means is a framework for building LLM applications. They share a name and nothing else.

So here is the problem with the advice everyone gives.

The check everyone recommends misses half the attack

Slopsquatting is the attack where a model invents a package name consistently enough that somebody can register it first and wait. The defence in every write-up, including the sensible ones, comes down to the same instruction: before you install what the agent suggested, check that the package is real.

That advice has a specific mechanical meaning. Ask the registry. If you get a 404, the model made it up. If you get a 200, the name is fine.

The 24 August paper puts the flaw in one sentence:

An attacker who has already registered the hallucinated name trivially bypasses a pure 404 check.

Its authors, Akash Raj and Sargam Sahu, built a two-layer detector precisely because one layer is not enough. Their first finding is the one that should end the existence check as a defence: half of the hallucinations their pipeline flagged were packages already registered on PyPI, sitting there as low-quality lookalikes of well-known projects. The deterministic layer missed all of them. A classifier trained on name and metadata caught them.

pil, faiss, tabula and haystack are their examples. That table above is what a 404 check tells you about all four: nothing.

The check fails in the other direction too

If it only failed one way you could treat it as a floor and move on. It fails both ways, and the second direction is why teams that try to automate this end up ignoring their own tooling.

A registry 404 does not mean the model hallucinated. Frameworks ship virtual modules, bundled subpackages and import aliases that are valid in source and have no standalone registry entry.

The clearest example came out of npm. When the researcher behind a large package-hallucination study could not get his npm findings through the registry’s own security channels, he took them to Socket at the end of April. Socket reviewed all 18 npm candidates and published the result in July.

The four highest-volume names among them were @ember/service, @ember/object, @ember/routing and @ember/controller. All four are valid Ember imports that resolve through ember-source. A model emitting @ember/service is writing correct Ember. A registry lookup for that name returns 404.

Socket also found ssh-keys was a real package and metro-evaluator was already security-held, and those six names came out of the exploitable set. The researcher folded the finding into version 2 of the paper as a false-positive class.

I ran the same check against my own machine and got the same result at small scale.

I pulled every package install command my agents proposed in August 2026 out of the local transcripts: 1,100 transcript files, 135 sessions, 121 install invocations, 34 distinct package names. Then I asked npm and PyPI about each one.

Exactly one name 404’d: argos-mcp. It is my own MCP server, linked locally with npm link. Never published. The single registry miss in a month of agent output was a false positive, and it was mine.

The interesting part was the noise. The script I wrote to mine the transcripts was crude in its first version, and it pulled three ordinary words out of the agent’s prose that happened to sit next to install. The registry approved all three.

installation is a real package, 23 versions, last touched in 2022. both is a real package, one version, 0.0.0, whose description is the literal string ERROR: No README.md file found!. And binaries returns 200 with zero versions published and no maintainer listed, a name registered in 2018 and holding nothing at all.

Three words that were never package names, waved through by the check that was supposed to catch fabricated package names. One real local package, flagged.

That is a 34-name sample on one developer’s laptop, so treat it as an illustration rather than a measurement. It reproduced both error directions anyway.

Switching models does not help

The reflex fix is to have a second model check the first one’s suggestion. The research is unusually direct about why that underperforms.

The study Socket reviewed is a replication of the USENIX Security '25 work on package hallucination, run against five current models: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro and DeepSeek V3.2. Nearly 200,000 prompts. Beyond the per-model rates, it looked for names that every model invents, and found 127 of them.

PyPI Security’s prohibited-name list and normalisation defences left 41 of the 109 Python names registrable, and Socket’s review took the npm side from 18 down to 12. That is 53 names which all five models produce and an attacker could still register, as of the April 2026 disclosure.

One registration. Four vendors’ users.

Three of those names are public in the write-up: css-color-stop, dns-sd and dom-ains. I looked all three up while writing this. Two of them are still free.

css-color-stop was registered on 28 July 2026, six days after Socket published the article that named it. What sits there now is version 0.0.0: one file, 53 bytes unpacked, no dependencies, no install script, no README, no description.

An empty package with nothing that executes is what a defensive registration looks like, and that is the likeliest reading. Somebody saw the disclosure and took the name so that nobody else could. It is worth sitting with anyway, because the mechanism is identical either way. A name travelled from a published security write-up to a claimed npm package in six days, and the only difference between the harmless version of that and the other one is what the publisher decided to put in the tarball.

That 53 is the number that matters, and no single-model study could have found it. Asking Gemini to check Claude’s homework does not help when both invent the same name. The 24 August paper measures the family-level version of this: when the primary model and its fallback come from the same family, roughly 84% of the primary’s failures recur on the fallback.

The number everyone quotes has expired

While I was reading round this, nearly every blog post and vendor page cited the same two figures: 19.7% of generated packages do not exist, and 43% of hallucinated names recur across all ten reruns. Those come from Spracklen et al., which circulated as a 2024 preprint and was published at USENIX Security '25.

The replication measured current frontier models at 4.62% to 6.10%. The spread between best and worst model collapsed from 16.5 percentage points to 1.48.

Quoting 19.7% in September 2026 describes a model generation that has been superseded. That cuts both ways, though. The rate fell and the shape of the risk got worse, because the models converged. When every vendor’s model invents the same 127 names, the attacker’s job stopped being probabilistic.

Read the caveats too, because the same study carries real ones. GPT-5.4-mini was tested at minimal reasoning effort and refused 32% of prompts, so its rate is computed over the compliant remainder. The measurements are one week in April against specific model snapshots. And agents that query the registry before suggesting a dependency behave differently from a single model answering a single prompt.

This is a preprint with a vendor correction folded into it.

What to actually check

Existence is the wrong question. These are the ones that separate the four names in my opening table from their real counterparts:

• When was the last release? Every impostor above is dormant: 2012, 2017, 2019. Every real one shipped in the last two years. Staleness plus a name your agent just suggested is the strongest cheap signal there is.
• How many releases, and how many maintainers? One release and no maintainer is not a project. binaries has zero versions and returns 200.
• Does the description match what you asked for? haystack says it searches C structures in process memory. You asked for an LLM framework. The registry told you the truth and the check threw it away by only reading the status code.
• Does the import name match the install name? import cv2 installs opencv-python. This mismatch is normal in Python, which is exactly why a lookalike hides in it comfortably.
• Commit the lockfile, and install from it in CI. This is the same argument as pinning a version not being the same as pinning a commit. A lockfile stops drift on packages you already vetted. It does nothing about the first install, which is the moment slopsquatting attacks.
• Treat an agent-proposed dependency like a pull request from a stranger. The pre-install checklist for MCP servers is the same ten minutes of work, aimed at a different registry.

None of that is new. It is the vetting that the npm playbook already taught us and that we skip when the suggestion arrives from something that sounds confident.

Which is the actual mechanism here. The model was never guessing at a package name. It produced the most plausible continuation, and a plausible package name looks exactly like a real one. That is the whole attack surface.

tabula is a better name than tabula-py. That is why the model reaches for it, and why somebody registered it in 2012 for something else entirely.

The registry will confirm it exists. It will not tell you the model was wrong.

That gap comes from one survey.

METR ran it between February and April 2026 across 349 technical workers: 87 software engineers, 71 researchers, 129 academics and PhD students, 48 founders and managers. The result is a median self-reported speed change of 3x, and a median self-reported change in the value of the work of 1.4 to 2x.

Same people. Same tools. Same week. One question about speed, one about value, and a factor of two sitting between the answers.

If you estimate software for a living, that gap is the whole problem.

The number everyone quotes is the loosest one

Three days earlier, the same organisation published the piece that explains the gap. Task substitution and uplift, by Tom Cunningham and Parker Whitfill, argues that there is no single quantity called “how much AI sped me up”. There are three, and they line up in an order:

uplift on old tasks ≤ uplift in value ≤ uplift on new tasks

Old tasks are the work you would have done in 2021. New tasks are the work you choose now, knowing an agent is available. Value is what you actually got.

Their worked example is deliberately small. An engineer splits an eight-hour day between writing documents and writing pull requests, an hour each. AI halves the cost of a PR and does nothing for documents. Measured on the old task list, that is a 33% uplift. Measured on the new one, where the engineer now writes eight PRs instead of four, it is 50%. Value lands somewhere between the two.

Widen the gap between the two activities and the bounds fly apart. In their more extreme case, where a PR goes from five hours to one, the three measures span +67%, +124% and +200%.

Three numbers, one engineer, one afternoon. All of them honest.

Here is what that does to an estimate. When somebody tells you their team is 3x faster, they are almost certainly reporting uplift on new tasks, because that is the number a human can feel. It is also the loosest of the three, which is the authors’ whole point.

They hedge it properly, and so should you: the ordering holds under simplifying assumptions, and their own footnote allows that it can reverse in practice. What it reliably tells you is which of the three questions you are being handed an answer to. The survey respondents asked about value instead of speed landed at 1.4 to 2x, and they were the same people.

Every respondent answered honestly. They were just answering a different question from the one you need answered when you commit to a date.

You are not doing the old work faster

This is the part that breaks estimation specifically, and it is worse than a measurement error.

An estimate is a comparison. You are saying this new thing resembles that old thing, and that old thing took a fortnight. The whole method depends on the two being commensurable.

Task substitution removes that. Whatever multiple you apply, you are applying it to a task list you would never have written down in 2024, because half of it was uneconomic.

The dashboard nobody asked for. The refactor across forty files. The second implementation you built to compare against the first. The prototype you threw away on Thursday, which cost four hours and would have cost four days.

Some of that is real value and some of it is motion. The point is that your denominator is gone. The old ticket and the new ticket share a title and nothing else.

I have written before that the aggregate speed number stopped meaning anything once developers started choosing, per task, where they wanted the help. This is the same finding one level up. Estimation broke for the same reason METR’s follow-up study broke: the population being measured kept quietly changing what it submitted.

Where the afternoon actually went

Take a real-shaped ticket. Add a filtered export endpoint to a reporting service that has been in production for three years. Estimated at two days before agents, and honestly, two days was about right.

Here is the same ticket, split into what the work is actually made of:

Writing fell by five sixths. The total fell by an hour.

Three of those five rows did not move at all, because none of them are typing. Understanding a codebase is reading and holding, and an agent that reads it for you leaves you holding nothing. Integration is negotiating with systems that already exist and already have opinions. Consequences arrive on their own schedule.

The row that moved in the wrong direction is review, and it moved because the thing being reviewed changed. Reviewing code you wrote is remembering. Reviewing code you did not write is reconstruction, and it is the expensive half of the job now. Four hours to review one hour of generated code is the actual price rather than a failure of discipline, and the reason so many teams post an improved cycle time is that they are not paying it.

Your rows will differ. Mine differ by codebase, and the ratio moves a long way depending on whether I know the subsystem. What holds across all of them is the shape: the one row that collapsed is the row that was never the majority of the estimate.

What I estimate now

I estimate the verification, not the generation. The question I ask about a ticket is no longer “how long to build this” but “how long until I can defend it”. Those were nearly the same number when I typed everything. They are not close any more, and the second one is the one that ships.

I estimate in unknowns, not in hours. How many things do I not know yet? Which of them can only be answered by touching production data, or by a conversation with someone who is on holiday? Generation did nothing to that list. It is the same list it was in 2019, and it still explains most of the variance in how long anything takes.

I do not re-estimate when the diff appears. This is the seductive one and the reason so many two-day tickets ship on day three. A complete-looking diff at 11am feels like evidence that the work is nearly done. It is evidence that one row of the table is nearly done. The unread diff is inventory rather than progress you can bank, and reviewing it is the part you had not started.

I record what I did, not what I would have done. Counterfactual self-report is the least reliable instrument in this whole discussion, and the survey says as much about itself. It puts the average overestimate of AI’s effect on time at 40 percentage points, drawing on METR’s 2025 trial, and warns that its own respondents are probably making the same error. That trial’s headline number is now flagged as out of date. The perception gap is the part that survived.

So I stopped asking myself how much faster I was. I write down what the ticket took, and I compare it to the estimate I made before I opened the editor. That number is boring and it is measured, which is precisely what makes it worth having.

None of this makes estimates good. Estimates were never good. What changed is that the one part of the job that used to be predictable, the sitting-down-and-typing part, stopped being the part you were estimating, and a lot of teams updated their commitments as though it still was.

So when someone tells you their team got 3x faster, believe them. They are reporting uplift on new tasks, accurately, about a task list that no longer resembles the one your roadmap was built on.

Then ask what shipped.

Then you try it on a real repository and hit the wall immediately.

The convention you correct most often is usually the one your codebase already breaks in three hundred places. You cannot switch that rule on as an error, because the build goes red on code nobody is touching today.

So you set it to warn, and a warning is where a convention goes to die. Nobody reads a stream of three hundred warnings. Your agent reads it as evidence that this codebase does the thing.

There is a second kind of guard for exactly this case, and almost nobody wires it up deliberately.

The rule you cannot switch on

A categorical rule says never. A ratchet says never worse than yesterday.

The mechanics are always the same. You run the check once, write the result into a file, and commit that file. From then on the check compares against the stored result and fails only on the difference.

What was already there is grandfathered. What the agent just added is not, and the number is allowed to move in one direction only.

PHPStan shipped this in October 2019 and described it in one sentence that is still the clearest statement of the idea: the baseline “allows you to be interested in violations only in new and changed code”. The documentation is blunt about the trade too. It works for a few dozen to a few hundred errors, and if you have fifteen thousand you should be fixing your configuration instead.

Three shapes

The frozen list. Every violation that exists right now, recorded by rule and file. PHPStan writes phpstan-baseline.neon, detekt writes a baseline.xml, and ESLint has shipped the same thing since v9.24.0 in April 2025:

# fix what can be fixed, freeze the rest
eslint --fix --suppress-all

That writes eslint-suppressions.json, and the docs tell you to commit it “so that the suppressions are shared with all the developers”. Two details matter more than the flag. Only rules configured as "error" get suppressed, so this is a mechanism for standards you actually mean. And detekt splits its file into CurrentIssues and ManuallySuppressedIssues, which is the distinction most teams lose: what is here because history put it here, versus what is here because someone decided it should be.

The single number. A per-file coverage floor, a bundle budget, a surviving-mutant count, type-coverage --at-least 98. One value, checked in, and a build that fails when it moves the wrong way. type-coverage even ships --update-if-higher, which tightens the stored threshold automatically whenever a run beats it.

The diff gate. Sonar does not store a list at all. It computes what counts as new code, and in a pull request analysis, measured against the target branch, “Only issues on new code are reported.” Nothing is grandfathered explicitly, because the diff decides what is yours.

Why this lands on an agent

An agent already self-heals against deterministic feedback. Give it a failing check with a clear message and it will fix the violation, then generate the next file closer to the pattern. That is the whole argument for custom lint rules.

A ratchet sharpens that message. The failure names the file the agent wrote thirty seconds ago and nothing else. There is no wading through inherited noise, no judgement call about which of the three hundred warnings are its problem, and no plausible reading in which the codebase’s own habits excuse it. The baseline is the codebase’s habits, in a file, already accounted for.

Wire it into the agent’s own loop. A check that runs after the turn ends gets fixed by the agent. A check that runs in CI ten minutes later gets fixed by you.

Four ways a ratchet rots

Regenerating the baseline. This is the one that kills the whole idea, and PHPStan’s own docs put it best: “The life goal of a baseline file is to not exist.” Regenerating is one command, it makes the build green, and it looks like maintenance in a diff. Treat a growing baseline as a reviewable event with a reason attached, the same way you would treat a skipped test.

ESLint handles half of this for you. When a suppressed violation no longer occurs, the run fails with There are suppressions left that do not occur anymore, so the file cannot quietly hold space for problems you already fixed. The other half, the file growing, is still a human judgement.

The agent editing the file. It will. A baseline is a text file in the repo, the failing check names it, and making it bigger is the shortest path to green. Put the baseline behind whatever your harness uses to make a path off limits.

In this repository that job belongs to a protect-paths.sh hook, which blocks writes to a fixed list of paths and hands the reason back to the agent. Until I wrote this post, the recorded audit runs under docs/standards/ were not on that list. They carried an instruction inside the file saying never to edit them, which is a convention, and a convention holds exactly as well as a convention holds. Checking my own claim is what moved them onto the block list, which is roughly how every rule in that hook got there.

Ratcheting the metric instead of the property. SpecBench, published in May 2026, opens on the reason this matters: “As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite.” Every frontier agent in their runs saturates the visible tests, the gap to held-out tests grows by 28 percentage points for every tenfold increase in code size, and one failure mode was a 2,900-line hash table that memorised test inputs.

So a coverage percentage is the worst thing you can ratchet, because the cheapest way to raise it is to write tests that assert nothing. Ratchet something with no cheap fake: surviving mutants, a count of one specific violation, a bundle size in bytes.

Nobody lowers it. A baseline that only ever gets regenerated is a budget with better manners. The fix is boring: date each run, keep it, and compare across quarters instead of across commits. The dated JSON files under docs/standards/ in this repo exist for that, one per run, never edited, so the number worth watching is what the score still says three months after the day everything got fixed.

What it buys upstream

Faros AI’s AI Engineering Report 2026 reached the conclusion this post is built on: “the code arriving for review is often not review-ready. Reviewers are not just assessing more code, they appear to be working harder to bring that code up to a standard it should have met before the PR was opened.”

A ratchet removes one class of comment from that queue permanently. It makes the standard a precondition of the pull request existing, so the reviewer never has to ask for it.

It only covers the half a machine can see, which is the same limit every instrument on this blog has. A green board says the checkable half is in order. It says nothing about whether the code does what the ticket asked, and that half still needs your eyes.

One last honest note. Betterer, the tool that generalised this pattern for JavaScript with a .betterer.results file and a smaller constraint, has not had a stable release since v5.4.0 in August 2022, and nothing at all has been published since an alpha in December 2024. The idea outlived the tooling. Learn the pattern and build it on whatever checker your stack already runs.

Start with one convention. Pick the one you corrected twice this week and cannot fix repo-wide, enable it as an error, generate the baseline today, commit it, and run the check where the agent will see it fail. The churn you are trying to avoid is mostly code written against a standard nobody enforced at the moment it was typed.

Yesterday’s code is a fact you have to live with. The next two hundred lines are still a choice.