You changed model on a Tuesday. By Friday something feels off.
The diffs are longer. The agent needed two attempts where it used to need one. A function came back with a shape you would not have written. Nothing is broken exactly, but you keep noticing things.
So you ask the obvious question. Did the model get worse?
Then you notice you have no way to answer it.
The best-documented case is one where the model never changed
In August 2025 people started reporting that Claude had got worse. The reports were vague, contradictory, and easy to write off as the thing that always happens once people get used to a tool.
They were right. Anthropic's postmortem found three separate infrastructure bugs running at the same time.
A context window routing error sent requests to the wrong server type. It started on 5 August affecting 0.8% of Sonnet 4 requests, a load balancing change on 29 August made it worse, and by 31 August it reached 16% of Sonnet 4 requests. Roughly 30% of Claude Code users in that window had at least one message routed to the wrong place.
A TPU misconfiguration corrupted output between 25 August and 2 September, occasionally dropping Thai characters into English responses.
An approximate top-k miscompilation in XLA:TPU, a mixed precision problem between bf16 and fp32, affected Haiku 3.5 for over two weeks.
Three bugs, overlapping, across three hardware platforms. The weights never changed.
Two things in that postmortem should bother you more than the bugs do.
Anthropic's own evaluations did not catch it. The stated reason is that "Claude often recovers well from isolated mistakes", so the damage averaged out at the aggregate level while individual sessions were visibly worse.
And the overlapping bugs produced "confusing and contradictory reports". Three real problems arriving together looked like noise to the people holding the telemetry.
If the vendor needed a month and privileged access to work out what you were feeling, your feeling was never going to get there on its own.
"The model got worse" is three different claims
Sort them, because they need different instruments:
- The weights changed. A new version shipped, or an old one was retired underneath a name you thought was pinned.
- The serving stack changed. Same weights, different routing, precision or hardware. This is the Anthropic case, and from the outside it is invisible.
- You changed. New codebase, harder tasks, more trust, less reading. This one is real and nobody wants it to be the answer.
Most arguments about model quality are two people arguing different claims from that list without realising it.
Re-running the prompt proves nothing
The instinct is to run the same prompt against both models and diff the results.
That does not work, and the reason is more interesting than "LLMs are random".
Thinking Machines Lab ran 1,000 completions of Qwen3-235B at temperature 0. They got 80 unique outputs. The first 102 tokens were identical in every single run. Divergence started at token 103.
At that first fork the split was lopsided: 992 runs went one way, 8 went the other. A thousand tokens later the most common completion appeared only 78 times. One rare token early on drags the rest of the generation with it, which is why a difference too small to care about at token 103 is a different answer by the end.
The folk explanation is concurrency plus floating point: threads finish in a nondeterministic order, atomic adds accumulate differently. That explanation is wrong, and they show it by running the same matrix multiplication a thousand times and getting bit-identical results every time.
The real cause is that the kernels are not batch-invariant. RMSNorm, matrix multiplication and attention each produce slightly different numerics depending on how many requests are in the batch. The server batches by load.
The variable that moved your output was how many strangers were hitting the API at the same moment.
With batch-invariant kernels, 1,000 runs produced 1,000 identical outputs, at roughly 1.6 times the runtime.
This gets misquoted in both directions, so be precise about it. Anthropic states plainly that they "never reduce model quality due to demand, time of day, or server load". That is a statement about policy, and nothing above contradicts it.
What moves is the arithmetic underneath the policy. Your bytes still differ between two runs of the same prompt on the same model, and the thing that moved them lives outside your machine and never appears in any log you can read.
Temperature 0 is not a controlled variable. You cannot compare two single runs. You cannot reliably compare one run against itself.
Asking a model which output is better
Second instinct: hand both outputs to a judge model and ask which one is better. Cheap, fast, and it fails in a way that is specific to this question.
Judging the Judges covers 15 judge models across MTBench and DevBench, 22 tasks, around 40 solution-generating models and over 150,000 evaluation instances. Position bias "is not due to random chance and varies significantly across judges and tasks". Swap which answer is A and which is B, and the verdict moves.
The finding that matters here is the second one. Position bias is "strongly affected by the quality gap between solutions", and position consistency drops when the two answers are close in quality.
Hold that against what you are actually doing. Comparing two versions of a frontier model on your own tasks is a close comparison by construction. The judge is least trustworthy in precisely the band where you need it, and rock solid in the band where you would never have had to ask.
The number usually offered in defence of judges is that they agree with humans about 80% of the time, roughly what two humans manage with each other. The largest evaluation of judges run so far says that number is partly an artefact of how it gets measured. Published in June 2026, it covers 21 judges from nine providers over 118 runs and roughly 541,000 individual judgments, including the April 2026 frontier. Exact-match agreement, the metric behind the 80%, "does not correct for chance and systematically overstates discriminative ability", and correcting for chance costs 33 to 41 percentage points on MT-Bench.
The result in that paper you should actually act on is what the authors call a consistency-bias paradox. Two production-deployed judges combined test-retest reliability above 0.95 with position bias above 0.10. Ask one of them the same question ten times and you get the same answer ten times, while the answer still moves depending on which candidate you listed first. Repeatability is the property you would naturally check to convince yourself a judge is sound, and it is the one property that cannot catch this.
Judges also lean towards their own family and towards text shaped like their own tuning. One of the two things you are comparing is usually the judge's relative.
The metrics are right, the sample size is wrong
Complexity, clone detection and architecture fitness functions do measure something real. What they describe is a body of code.
They cannot tell you anything about one pull request. A cyclomatic complexity figure on a single diff arrives with no error bar. It turns into evidence across three months of merged work, which is the lesson churn already taught: watch the slope.
These instruments answer a question about the last quarter. On the Friday you got suspicious, they have nothing to say.
What actually works
Measure the outcome on a fixed corpus, with a scorer that has no taste.
- A frozen task set. Twenty to fifty tasks pulled out of your own repo history, along with the diffs they produced. Your problems, at your own level of mess.
- A scorer that cannot hold an opinion. Does it compile. Do the existing tests pass. Does the mutation score hold. Does the dependency rule still hold. Every one of those is a boolean or an integer that no language model produced.
- Enough runs that the noise cancels. Given 80 unique outputs in 1,000, one run per task is an anecdote. Five per task per model is where it starts to mean something.
- A baseline recorded before the swap. The one part of this that cannot be reconstructed afterwards, and the part everybody skips.
That is a small, boring harness. It also costs real money. Fifty tasks, five runs and two models comes to five hundred agent sessions before anyone reads a result, which is its own argument for running the cheapest model that can do the job.
The 2026 governance literature arrives at the same place from the opposite direction, by treating a model update as what it actually is: a dependency bump nobody authorised. "Test Before You Deploy" frames it as a software supply chain problem, and argues that regression testing and version pinning do not on their own give the deployer anything to hold during "opaque model evolution". What it proposes instead is a contract for how the model is allowed to behave, a test suite organised by deployment risk rather than by feature, and gates that block an update until it clears them.
Its exploratory finding is the one Anthropic ran into from the other side: targeted testing in specific risk areas "can uncover performance regressions that overall metrics miss". And the open problem it names, how to set reliable thresholds in a non-deterministic system, is this whole post in one line.
Build it while you are calm
Vibes are the industry default and I understand why. The harness costs a weekend, the model moves again in six weeks, and the honest reward for the whole exercise is usually "no measurable change".
The alternative is the position everyone was in during August 2025. A real degradation, a strong feeling about it, and no way to separate that feeling from having simply got used to the tool.
So build the instrument before you need it. By the time you need it you are already suspicious, and a suspicious engineer reading unlabelled diffs will find exactly what they went looking for. A frozen task list and a dumb scorer is not much of a laboratory. It is enough to turn "this feels worse" into a number, and a number is the only version of that sentence anyone can argue with.