Sonnet 5 Tokenizer Inflates Token Counts 42% on English Text

The retrieval layer is the part worth acting on, not the leaderboard fight. NVIDIA's Nemotron 3 Embed 8B tops RTEB at 78.46 NDCG@10 and 75.45 on MMTEB Retrieval, and its NVFP4 variant claims ~2x BF16 throughput on Blackwell at >99% quality retention. Retrieval infra is isolated and reversible, so the test is cheap: if that throughput number holds on your own eval set, embedding cost drops roughly in half with negligible recall loss. That is a one-day spike entirely within your control.

Kimi K3 is harder to act on. Moonshot's 2.8T-total / 50B-active MoE runs under 2% activation, 16 of 896 experts. It posts an AA Intelligence Index of 57, line-ball with Opus 4.8's 56, at $0.94 per task versus Opus's $1.80, and ranks #1 in Frontend Code Arena at a 76% win rate. It cut output tokens 21% for +13 index points over its predecessor. Open weights land July 27.

The thing the headline index doesn't tell you: on AA-Omniscience, accuracy climbed to 46% from 33% while hallucination worsened to 51% from 39%. Capability scaled and calibration regressed. The model got more confidently wrong on exactly the axis a single leaderboard number won't surface, which is the axis that hurts in production.

Two methodology flags temper the rest. ProgramBench author Ofir Press notes K3 used a non-recommended averaging metric, and Bindu Reddy warns the numbers may be inflated absent uncontaminated evals like LiveBench. Self-hosting is off the table for most: a 1.58-bit quant exceeds 512GB, and you'd need 64+ accelerator supernodes plus a vLLM build patched for Kimi Delta Attention, which breaks conventional prefix caching. The pattern across sources is consistent. Frontier capability is now cheap and open-weight, so the eval harness and verification layer are what actually ship safely.

The uncomfortable finding underneath the pricing story: a May 2026 paper showed tokenizer ambiguity permits 50.85% billing over-reporting below detection thresholds, with hidden-reasoning inflation reaching 1,469% in the most permissive setting. The vendor controls the unit it bills you on. You cannot verify that unit from invoices alone.

That reframes Sonnet 5. Anthropic's new tokenizer emits ~30% more tokens by its own estimate. Independent testing measured 1.42x on English, 1.33x Spanish, 1.27x Python, ~1.0x Mandarin versus 4.6. The price card ($2/$10 promo, $3/$15 after Aug 31) is unchanged, so effective English cost lands at $4.26/$21.30 per million old-token-equivalents. That is a 42% rise that reads as flat. The thing the price card doesn't tell you is which language you run.

The response is layered. Own-corpus re-benchmarking on cost-per-completed-task, segmented by language and content, is the only number that survives contact with production. An independent token meter counts input client-side, logs output/tool/retry tokens, and reconciles monthly. Failover becomes resilience: a Commerce recall disabled two frontier models globally in ~90 minutes for 14 days, so a pre-validated open-weight fallback is now a production requirement. Open-weight models already handle 29% of Vercel's token volume at under 4% of spend.

Agent economics amplify all of it. Cost is heavy-tailed. A few retry storms and reasoning escalations dominate the bill, so forecast on p95, not the mean, and cap the tail. Uber exhausted a full-year Claude Code budget in four months and capped spend at $1,500/employee/month. Multi-model orchestration (Sakana's Fugu-Ultra) hit 73.7 on SWE-Bench Pro versus 69.2 for Opus 4.8 alone. That gap is the research leaderboard, not your invoice, but it points the same direction: value is moving above the model layer, which breaks the habit of benchmarking one model at a time.

These findings share a root cause worth naming before the numbers: improving an input by a human-intuitive metric can move it off the model's training manifold. Whisper was trained on noisy, in-the-wild audio. So denoising before transcription doubled word error rate — 10.53% to 21.66% on base. The thing a clean-audio stage doesn't tell you is that it introduced a train/serve distribution mismatch, and that mismatch halved accuracy.

The second confounder invalidates evals directly. LLM-as-judge verdicts flipped at a rate that swung from 0.6–3.1% to 28–41% based solely on turn-taking order. If the offline eval or preference-labeling harness doesn't randomize order and score symmetrically — A-vs-B and B-vs-A — position bias is a dominant, uncontrolled confounder. That is enough to reorder rankings and poison tuning data.

The positive counterexample is instructive. A Brown economist caught mass LLM cheating with no AI-content classifier. A class average jumped from a historical 65–80 baseline to 96, a multi-sigma anomaly. Then 22 of 27 students who aced the take-home no-showed the in-person final, where the average collapsed to 48. Baseline-drift monitoring plus a control arm beat any 'this-is-AI' detector. It's the monitoring most teams already own but never point at their human-labeled streams.

Related, and cheap to check: verifier-bound RL runs at ~5% mean FLOP utilization when rollouts stall on a slow reward source, and chain-of-thought collapses into repetition during reward hacking. Score final outputs against ground truth, never CoT plausibility. Each of these is a days-long audit against an existing harness, not a research program. Every one can be quietly costing accuracy or compute right now.