Debugging with a coding agent: give it the search, keep the hypothesis
7m read time

Debugging with a coding agent: give it the search, keep the hypothesis

An AI debugging workflow for Claude Code and other agents: what the fault-localisation research shows agents miss, and how to split the work so the hypothesis stays with you.

The biggest single frustration developers reported in the 2025 Stack Overflow survey was AI output that is almost right, but not quite. Sixty-six percent of them. The second biggest, at 45%, was that debugging AI-generated code takes longer than debugging your own. Trust in the accuracy of AI tools fell from 40% to 29% in a year, with 3% saying they highly trust it.

So the profession's loudest complaint is about debugging, and the thing being sold to fix it is another agent.

And yet. Yesterday an agent found a null in a serialiser in about ninety seconds, in a file I had been staring at for twenty minutes. It was right. It was right in a way I would have got to eventually, by the boring route.

Both of those things are true, and the useful question is which half of debugging you are handing over.

What an agent is genuinely better at

Breadth, and stamina for the boring part.

A stack trace that touches forty files is tedious for me and free for the agent. Finding every call site of a function, checking each one against the contract, reading a dependency's source instead of guessing at its behaviour: that is real work I skip when I am tired, and it does not skip it.

It also has no ego about the code. I defend my own architecture without noticing. It reads my clever bit as neutrally as it reads everything else, which is occasionally humiliating and usually useful.

That is the mechanical half of root-cause work. It is worth handing over. The question is what happens to the other half.

Where AI debugging actually fails

There is a study that answers this better than any blog post could, including this one.

A team ran 750,013 fault-localisation tasks across ten models and more than 1,300 real Java and Python programs (arXiv 2504.04372). They injected faults, kept only the programs where a model successfully found the fault, and then applied semantic-preserving mutations: changes that alter nothing about what the program does.

Dead code. Misleading comments. Misleading variable names. Reordered functions.

The models then failed to find faults they had already found in 78% of cases.

The breakdown by mutation is worse than the headline:

Mutation appliedLocalisation accuracy retained
Misleading variable names29.02%
Misleading comments25.63%
Dead code injection20.38%
Function reordering (Java)17%

Read that list again as a description rather than an experiment. Dead code. Comments that no longer match. Names that lie about what the thing does. Functions in the order they were bolted on rather than the order they run.

Those mutations describe any codebase that has been alive for three years.

The same study found the models detect 56% of the faults they catch in the first quarter of the code, and 6% in the last quarter. The bottom of your file is barely searched. And the newer Claude and Gemini releases they tested moved fault localisability by one to two percent, so this is not a wait-for-the-next-model problem.

Scale does its own damage. LinuxFLBench put agents on 250 real Linux kernel bugs across a tree of 69,000 files and 28 million lines (arXiv 2505.19489). The best performer, SWE-Agent, hit Recall@1 of 0.416.

The paper's sharper point is the delta: those are the same agents that score 16.7% to 31.9% higher on ordinary-sized software. Up to a third of the accuracy goes to size alone.

And on the bugs where you most want help, regression bugs, the numbers start out grim. On RegMiner4APR, 99 regressions from 32 repositories, classic automated repair tools scored 0%. The best LLM approach produced correct patches for 9% (arXiv 2506.13182).

Then the researchers told the model which commit introduced the bug.

Sixteen percent. Nearly double, from one fact that was sitting in git log the whole time.

Every one of those failures is an information failure

That is the thread running through all three papers, and it is the whole basis for a workflow.

The model has reasoning to spare. What it lacks is the narrowing facts you already hold: which commit, which subsystem, which of the names in this file is currently lying. Give it those and the numbers move. Withhold them and you get confident output pointing at the top of the wrong file.

Which gives a division of labour that has nothing to do with how hard the bug is.

You narrow and hypothesise. It searches and tests.

The handoff protocol

Reproduce it yourself first. A bug you cannot trigger on demand is a bug you cannot verify a fix for, and the agent will happily hand you a patch that makes the symptom go away. If the reproduction takes an hour, that hour is the work, and it is not delegable.

Narrow the search space before you hand over. Name the subsystem, the module, the layer. The kernel study's own improvement came from constraining the search space. You know which half of the system this lives in, and saying so is worth more than any phrasing trick.

Hand it the diff along with the error. git log -S, the last green deploy, the commit that touched this path. That is the 9% to 16% jump in the paper, and on a regression you can usually find that commit yourself.

Ask for a hypothesis before a patch. This one has support from an unexpected direction. Claude Code's own EnterPlanMode tool description, in the 2.1.224 binary, lists this under the conditions that require planning:

Unclear Requirements: You need to explore before understanding the full scope

  • Example: "Fix the bug in checkout" - need to investigate root cause

Bug fixing with an unknown cause is classified as a planning task by the tool itself. The same description then tells the model to skip planning for "obvious bugs, small tweaks", and obviousness is precisely the judgement the mutation study shows it makes badly. So make the call yourself: plan mode before an edit, and approve the reasoning rather than the diff.

Make it stop after three. The systematic-debugging skill I run has an iron law at the top, NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST, and a rule I have come to value more: after three failed fixes, stop and question the architecture instead of attempting a fourth. Three failures in a row is information. It usually means the bug is not where anyone is looking.

Keep the session short. Debugging is the fastest way to fill a context window: logs, stack traces, whole files pasted in to check one line. By the time you have hypothesis four you are reasoning in a session where the early evidence is buried, which is its own failure mode. Restate the problem in a clean session more often than feels necessary.

What I do not hand over

Bugs where the real question is what the code was supposed to do. The agent can tell me what the code does. It cannot tell me what the invoice was meant to say, and a confident answer there is worse than no answer.

Races, timing bugs, anything with poor observability. Limited observability is one of the three reasons LinuxFLBench gives for why kernel debugging breaks agents. It breaks them everywhere else too.

And anything in code I have never read. Which is the part that took me longest to be honest about.

The instinct you have to keep

The failure mode in that study is a model producing a fluent, specific, well-argued answer about the wrong function, because a variable name told it a story.

You cannot catch that by reading the patch. The patch will look fine. You catch it by knowing roughly what that module does before you asked, which is a skill with a maintenance cost, and it decays quietly while your throughput goes up. Reading code deliberately is the only thing I have found that keeps it topped up.

Keep some bugs for yourself. A steady few, in the systems you own.

The 45% who say debugging AI code takes longer are not bad at prompting. They are debugging code they never read.

The handoff works when you did the reading.

(28 of 28)
01Getting the best out of Claude Code02Superpowers: teaching Claude Code to think before it types03Claude Code hooks: deterministic control over AI workflows04The CLAUDE.md file: give your AI permanent memory05Stop asking your agent nicely06What's new in Claude Code: notes from the London talk07The best number in Opus 4.8 isn't a benchmark08Stale memory is worse than no memory09The agent is just a loop10Build an MCP server, then ask whether it should exist11Skill, subagent, hook, or slash command? Pick the right one12Log in to MCP servers from your shell13The day 'default' became 'Manual'14How to write a proper Claude Code skill15How to write a proper Claude Code subagent16Claude Code permissions: the guide I wish the docs were17Sandboxing Claude Code: put your agent in a box that holds18Prompt injection defense for developers who ship agents19Which Claude model for which coding task20Refactoring legacy code with a coding agent: start with characterization tests21MCP server authentication: OAuth, scopes and rate limits22Opus 5 is here and your effort settings just expired23AI agent incident response: what to do when your coding agent goes wrong24Claude Code /doctor: the health check became a context audit25Claude Code context management: when to /clear and when to /compact26Git worktrees for parallel coding agents: what they isolate and what they share27Claude Code plan mode: decide before the agent writes28Debugging with a coding agent: give it the search, keep the hypothesis