Two instruments looked at the same code and came back with opposite verdicts.
New Relic's 2026 State of AI Coding report, published on 10 June, found that 94% of technology leaders rate AI-generated code as higher quality than human-authored code at the moment it is reviewed. Among the same 200 respondents, 82% had suffered at least one production failure tied to AI-generated code in the previous six months.
Review said it was good. Production disagreed.
Which instrument was wrong
Neither. They measure different things, and the gap between them is the subject.
Review reads intent. A reviewer looks at a diff and asks whether this is a sensible way to do the thing it says it does. That is a question about the code as written.
Production reads behaviour. It asks what actually happened when this ran against real input at real concurrency with a real dependency having a bad afternoon. That is a question about the system.
Those two have always been different questions, and what changed is the correlation between them. When a human wrote the code, sloppy thinking usually showed up as sloppy code, so review caught a useful share of the behavioural problems by proxy. AI-generated code broke that proxy. The AI code review bottleneck quoted Faros AI on why: the code is "often superficially convincing: idiomatic, well-named, stylistically consistent with the surrounding codebase", with the structural failures underneath.
Idiomatic and well-named is exactly what review is tuned to reward. So the instrument that used to catch the problem by proxy now reads clean, and 94% of leaders sign off on it.
The numbers in between are consistent with that. 78% report a measurable rise in production incidents tied to AI code, 86% report more senior-engineer firefighting, and 74% say at least a quarter of their AI-generated code needed significant rework over the past twelve months. Faros' own read of that research, published 6 August, lines it up against their telemetry: incidents-to-PR ratio more than tripled, bugs per developer up 54%.
The trade you are making
Instrumenting is a net under the part of review you already stopped doing. It is a worse net than reading would have been, because it catches things later, and later means after a user found it.
The same survey makes the limit obvious. 96% of those leaders rate observability as very or extremely important for managing AI-generated code, and the report notes that most organisations in the sample already run an observability stack, many of them more than one tool. 82% had a production failure anyway, and instrumentation prevented none of them. What it does is shorten the distance between a failure existing and you knowing which failure it is.
Worth saying where the figures come from. New Relic sells observability, and the report treats investing in it as settled, leaving only the question of which platform you standardise on. That is a sales conclusion drawn from real survey data, and the 82% is the more interesting half of it precisely because it undercuts the pitch.
Faros reads the same diagnosis and arrives somewhere else, arguing you should address code quality at the authoring stage rather than treat symptoms downstream. Do that. It is cheaper, and it is where the problem lives. This post is about the share that gets past it anyway, because 62% of teams in that survey already ship AI-generated code without line-by-line verification, that number is not going down, and the alternative to a net is no net.
Logging is not observability, and the difference is a question
The word comes from control theory, where Rudolf Kálmán defined observability around 1960 as a property of a system: how well you can infer its internal state from its external outputs. Charity Majors carried that definition into software, and the useful half of it is this.
Logging answers the questions you had when you wrote the logging.
Observability is being able to ask a question you did not have.
That distinction sounds academic until you go and read your own instrumentation, which is what I did.
What my own logging cannot tell me
This site runs an Express server on a host I administer myself: static releases, an atomic publish and rollback, and an AI chat endpoint backed by Gemini. It has had structured logging since long before I wrote this post. server/utils/chatLogger.mjs appends one JSON line per event with timestamp, IP, language, message, response, duration, status, and a prompt-injection detector that flags abuse keywords and refusal patterns.
That is decent logging. It is also almost useless for the questions I ended up wanting to ask.
The live log sits on the deploy host rather than in the repository, so what follows is an audit of the instrument rather than a pull of the data. Four questions, none of which the code can answer.
Where did the time go? In server/services/ai.mjs, startTime is set on line 9. The knowledge base loads on line 14. So the duration I log covers reading every content file off disk, searching the blog index, the Gemini call itself, and any retry sleeps, added together and reported as one number. If p95 latency doubles tomorrow I know that it doubled, but not which of the four moved.
Did it retry? The call sits in a loop of up to three attempts, sleeping one second then two on a 503 from Gemini. A request that failed twice, waited three seconds and succeeded on the third attempt logs status: 200 and nothing else. An upstream having a bad hour and a user writing a long prompt produce the same log line, distinguishable only by a duration number that, per the previous paragraph, is already blended.
Who was this? There are three event types in that file: chat, security_block, rate_limit_block. Three different sets of fields, and the only thing all three share is an IP address. That is enough to guess at a visitor and nowhere near enough to follow a request. When a rate-limit block and a chat row carry the same address a minute apart, nothing tells me they were the same person, and nothing at all ties a block to the request that provoked it.
And the one that actually stings. The knowledge base is filtered by relevance: loadBlogIndex searches the blog index against the visitor's message, and when it matches nothing it returns an empty string. The prompt is then assembled without the blog section, silently, and the model does what the system prompt tells it to do when the knowledge base lacks something, which is to say it does not have that information. Nothing logs the retrieval miss.
Then the abuse detector reads that answer. Its refusal test is a list of substrings including sorry, unable, cannot, helaas and kan niet. An answer carrying any of those sets isRefusal: true, and that flag on its own sets isAbuse: true.
Now take an injection attempt the keyword list does not recognise, something like instructing the model to answer only in French from here on. It gets politely declined, and it produces exactly that row: refusal flagged, no abuse keyword. A search returning zero results produces the same row.
The prose fields still hold the difference, so I can tell the two apart by reading them one at a time. Every structured field is identical, so I cannot filter, count or alert on which of the two I am looking at. The field that would separate them, whether the search returned anything at all, is the one nobody thought to write.
Every line involved is correct, passes review, and does exactly what it says.
That is the post in one bug: code whose behaviour you cannot ask about. I would never have found it by reading the diff, because I wrote the diff and it looked fine.
One event per request
The fix for all four is the same shape, and it predates the current vocabulary. Stripe published canonical log lines in July 2019: alongside normal traces, every request emits one long line at the end carrying its key characteristics. Jeremy Morrell's practitioner's guide to wide events puts the modern version plainly: for each unit of work, emit one event with all the information you can collect about that work.
One row per request. Wide, flat, boring. For the endpoint above that is roughly:
{
request_id: 'req_8f2a...', // joins every row about this request
route: '/api/chat',
lang: 'nl',
status: 200,
duration_kb_ms: 84, // the four numbers that were one
duration_model_ms: 1310,
duration_retry_sleep_ms: 1000,
duration_total_ms: 2394,
model: 'gemini-3.1-flash-lite',
attempts: 2, // the retry, no longer invisible
upstream_status_first: 503,
kb_posts_matched: 0, // the retrieval miss, now a fact
kb_chars: 41208,
history_length: 3,
response_chars: 118,
flagged_refusal: true,
flagged_abuse_keyword: false,
}Nothing clever is happening. The four unanswerable questions became four fields, and the request id makes the security and rate-limit rows joinable to the chat row instead of sitting in the same file pretending to be related.
Two rules make it work. Every field has to be on the same row, because the moment a fact lives one row away you are back to correlating by timestamp and hoping. And the row has to tolerate high-cardinality fields, meaning request ids, user ids, build shas and feature flags, because the question worth asking is almost always about one specific thing that went wrong.
Both of those rule out counters. A metric can tell you refusals went up. Only the wide row can tell you they went up because kb_posts_matched was zero on all of them.
OpenTelemetry is where this lands as a standard, and its span attributes are the same idea with a wire format attached. Start with the shape. A JSON line per request in a file you can query beats a half-configured collector, and it is roughly forty lines of work.
Instrument what you cannot verify
The choice of what to put on the row is where the judgement lives, and the useful filter is verifiability rather than importance.
Anything you can prove by reading the code does not need a field. Anything that depends on the state of the world at runtime does. Which upstream answered, how long it took, whether the cache hit, how many results the search returned, which release sha served the request, which branch of the conditional actually ran.
That maps onto AI-generated code almost too neatly, because the parts an agent gets wrong are the parts that touch the world. The pure function it wrote is probably fine. The retry policy, the fallback, the empty-result path and the error branch are where it guessed, and they are guesses you cannot check by reading, because reading is how they got approved in the first place.
So instrument the guesses. Every silent fallback gets a field naming which path ran, every empty result gets a count, every retry gets an attempt number. Those are the dull parts of the code, which is why nobody reviewed them carefully and why they page you.
This is the runtime end of a question measuring AI code quality asked at the other end. Complexity, duplication and architecture fitness functions all run without deploying anything, and they tell you what the code is. What it did is a measurement you can only take after it has run.
The trap at the end
One more figure from that survey, and it is the one I would put on a wall. 78% of teams now frequently or always prompt their AI tools to include logging hooks, span attributes and custom metrics in the generated code.
Read that next to the 62% who ship without line-by-line verification and the shape is hard to miss. The net under the code nobody read is being written by the thing nobody read.
Agent-written instrumentation is usually neat, well-named and stylistically consistent, which is the whole problem with everything else in this post. An agent will happily log that a function was entered, because that is inferable from the code and therefore easy to write. It will not know that your knowledge-base search silently returns nothing, because that fact does not exist in the diff. It exists in what happens next.
Telemetry that only records what the code says it does is a mirror, not an instrument. It will agree with the review, at length, in production, forever. Deciding what a system needs to be able to tell you is the same judgement as deciding whether its logic is right, and the pipeline was already the bottleneck before we started asking it to grade its own homework.
I went looking for what my own logging could not see and found two very different events filed under one flag, with no field anywhere to tell them apart. The gap had been there for months, through every review I ran, in code I wrote myself. The instrument was fine. I had just never asked it a question it was not built to answer.