Opus 5 Tops Intelligence Index With 50% Hallucination Rate

The harness, not the model, is the failure locus

The cleanest case in today's material is a paper surfaced by Techpresso. A celebrated long-document recall method reported 100% on tasks constructed to be impossible - because the gold answer was reachable from the model's own input. Corrected for the leak, the method does not work. No ablation study would have caught that. The defect lived in the evaluation data, not the architecture. The heuristic that transfers: when a hard-recall benchmark returns a near-perfect score, suspect leakage before you suspect a breakthrough.

Turing Post's read of a 2026 nine-system study of Chinese-language factual search adds the other half. Three axes, all routinely unreported:

• Coverage. Conditional accuracy across all nine systems clustered in a narrow 73.2-78.9% band. Abstention rates diverged sharply. Search engines answered nearly every query while some LLMs declined heavily. Conditional accuracy is computed over answered items only, so a single accuracy number pays a bonus for dodging. The thing that number doesn't tell you is coverage. What you need is a coverage-accuracy curve and comparisons taken at matched coverage.
• Class polarity. All nine systems performed worse when the correct answer was 'No'. Negation and refutation are systematically harder. A test set that skews affirmative-true produces a headline number optimistic by a margin nobody has measured.
• Controls. The study was cited as evidence that local Chinese AI had "surpassed Google," and Google was never in the comparison. No baseline, no claim. The same standard an internal "we beat vendor X" deck should meet on queries, evaluator, and corpus parity.

The same defect, wearing a growth-metric costume

The confounded-metric pattern repeats in every reported product win, each one instructive precisely because it was published as proof of impact.

None of these are causal estimates as reported, and all four would survive a strategy review unchallenged. Correlation with a good outcome reads identically to having caused it. The cleanest way to separate them is to run the counterfactual before you cite the win.

Two fixes worth engineering time

For temporal leakage specifically - models retrieving post-cutoff answers during agentic-search evals and forecasting backtests - AINews flags GenReasoning's BackSearch, which queries the web as it existed on a given date. Date-pinned retrieval is the cleanest available fix for backtests that currently score themselves against the future. Pilot it on one backtest before trusting any web-augmented eval number.

The deeper shift is what the eval measures at all. For agentic and transactional systems, answer accuracy on a static QA set says nothing about whether a multi-step task completed. Intent recognition, task-completion rate, option quality, and reversibility are the axes that decide production behavior, and none of them appear on a QA leaderboard.

Caveat on the source material: the nine-system study is unnamed with no disclosed sample size or evaluator. Treat its numbers as directional and its methodology lessons as durable.

A leaderboard that reports accuracy without coverage is ranking which system is best at declining to answer.

What "output constraints as infrastructure" actually buys

The interesting part of Netflix's in-house LLM stack is not build-versus-buy. It is where the constraint lives. Structured decoding, schema validation, and guardrail enforcement sit in the serving layer next to engine selection, model packaging, API surface, and deployment. Not in each consuming application. That placement kills drift. When every consumer writes its own JSON validator and its own refusal handling, the definitions diverge quietly, and each divergence is a separate incident. Centralize them and the contract becomes testable in one place, versionable with the model.

Be clear about the evidence: this is a narrative, not a benchmark. No latency, cost, or reliability numbers accompany the claim, and Netflix-scale engineering is the implicit prerequisite. The transferable idea costs far less than the stack does.

The bias finding that invalidates a common audit design

The item that changes how you evaluate models is the research finding that AI hiring models can manufacture their own novel biases rather than only inheriting stereotypes from training data. If disparities emerge de novo inside the model, data-level debiasing is necessary but not sufficient, and any fairness audit that stops at the training set is structurally incomplete. The check that measures the actual bottleneck is disparate-impact testing on held-out demographic slices of the model's outputs. Most teams skip it precisely when the training data looks clean.

This rhymes with the eval failures elsewhere in today's picture: a plausible input-side assumption stands in for an output-side measurement. The finding itself is directional - no sample size or effect size is reported - so treat it as a reason to add the test, not as a calibrated estimate of how large the effect is.

The caching pattern is the cheapest thing here

DoorDash's Entity Cache pairs Envoy with Valkey and reports 1.5M RPS at seven-nines availability with zero application code changes for adopting teams. The throughput is not the point. "Seven nines" arrives with no measurement window, so it is inspiration rather than a target. The transferable property is the zero-code-change adoption path, which is what decides whether a caching migration finishes at all. That maps onto read-heavy ML paths: feature fetches, embedding lookups, cached inference results, where a transparent proxy absorbs repeat traffic without touching a training or serving repo.

For agent work, the twelve-principle reliability rubric published under Apache-2.0 is a design-review artifact you can adopt in an afternoon. It will not make an unreliable agent reliable. It converts a vague reliability argument in a design doc into a checklist someone can fail.

Data-level debiasing is table stakes, not a finish line - if a model can invent its own biases, only output-slice auditing will ever see them.