~/blog/tag/testing
Testing
The suite is the only thing standing between a confident agent and production. What it has to be worth for that to hold.
What I write about here
Generation got cheap, so the test suite quietly changed job. It used to be a safety net under code somebody had already reasoned about. Now it is often the only thing that reasoned about the code at all.
That puts weight on tests they were never designed to carry, and it exposes how little most suites actually assert. A passing suite tells you the code ran. Coverage tells you a line was executed. Neither tells you anything would fail if the behaviour changed, which is the only question that matters when the author was a model that will happily write a test asserting what the code already does.
Mutation testing is the instrument that answers it, and it is the one I keep coming back to here. Break something on purpose, see whether the suite notices. The number that comes out is usually lower than people expect and always more useful than coverage.
The other posts under this tag deal with the surrounding problems. Characterization tests, because pinning existing behaviour is the precondition for letting an agent touch legacy code rather than a nice-to-have. Baselines that grandfather what already exists and refuse what was just added. Frozen task sets, for when the question is whether a model change made your output worse and re-running the prompt cannot tell you.
The argument for why any of this matters is in the AI and code quality guide. The code-quality series puts the method posts in reading order.
best entry points
- AI writes the tests. Mutation testing checks if they work.
The clearest demonstration of the problem. An agent writes the tests, they pass, and mutation testing shows how many of them would notice if the code broke.
- Refactoring legacy code with a coding agent: start with characterization tests
Where tests stop being hygiene and become the precondition. You cannot point an agent at code whose behaviour nobody has written down.
- Measuring AI code quality: the dashboard beyond coverage and mutation testing
What to watch past the mutation score, and the trap inside each instrument that makes a number improve while the codebase gets worse.
Code quality guards for AI code: the whole setup, from hook to CI
The full quality setup for a project where an agent writes code: what belongs in Claude Code hooks, pre-commit, pre-push, CI and the weekly mutation run. And the one property every guard needs, which is that it can fail, with the evidence from three of my own repositories.
readYour upload validator passed. Your server still got owned.
Bricksforge checked that an upload was an image, then trusted the browser to decide where its contents ended up. The lesson is in the handoff: a correct check can protect the wrong part of your workflow.
Empty catch blocks in AI code: the comment that silences your linter
AI-generated code swallows errors in catch blocks that hold nothing but a comment, and ESLint's recommended config waves them through because of that comment. What I found in my own repositories, why a model writes it, and the rule that sees through the comment.
What a dependency graph finds that your coding agent misses
I pointed a scanner at an internal app that agents had been working in for months. It took under two seconds and produced four tickets. Not one of the defects was in a file, which is exactly why nothing reading files had found them.
Quality ratchets for AI code: baselines that only tighten
A lint baseline lets you enforce a standard you cannot fix today. Existing violations are grandfathered, new ones fail the build, and the file can only shrink. How PHPStan, ESLint, detekt and Sonar do it, and the four ways a ratchet quietly stops working.
Did the model get worse? Comparing coding models without a benchmark
You swapped models, something feels worse, and you have nothing to point at. Why re-running the prompt and asking a judge model both fail, and the small boring harness that answers the question.
Measuring AI code quality: the dashboard beyond coverage and mutation testing
Coverage proves a line ran, mutation testing proves a test would catch a bug, and neither tells you if the code is quietly getting harder to change. What complexity, clone detection and architecture fitness functions actually catch.
Refactoring legacy code with a coding agent: start with characterization tests
A coding agent is at its best in the code nobody wants to touch, and most dangerous there too. How to refactor legacy code with AI safely: pin the behaviour first, then let the agent move it.
AI writes the tests. Mutation testing checks if they work.
AI-generated tests hit high coverage in seconds, but coverage only proves a line ran. Mutation testing proves the tests would actually catch a bug, and Chaos-MCP puts that loop inside your agent.
You can't spot the bug if you didn't write the code
AI isn't making you a worse programmer. It's making you a worse reader. On debugging instincts, skill atrophy, and why the biggest gap isn't in writing code but in understanding it.