OpenAI Pulls SWE-Bench Pro After 30% of Tasks Found Broken

The arithmetic is unforgiving. Agent tasks fire many sequential calls — planning, tool use, analysis — with accumulating context, but the prompt cache is keyed to the model. Route call #7 to a cheaper model and the entire context re-bills cold, forfeiting the ~90% discount. The counterintuitive result: a cache-hot expensive model can cost less per task than a cache-cold cheap one. Per-call cheapest-model routing, the default in most agent stacks, is a cost regression that no per-call price sheet will show you.

The production fix is a 4-stage pipeline that evaluates every request but commits the model once per session:

1. Guardrail filter screens requests.
2. Tiny router model — Arch-Router, 1.5B on HuggingFace, trained on human preference data (what developers actually pick per task type), routing preferences in plain English. The size constraint is the point. Routing that costs as much as the call it routes saves nothing.
3. Selection policy picks by cost or speed from dynamic pricing catalogs, keeping a runner-up on standby.
4. Model affinity — session-ID pinning preserves cache warmth across the multi-call task.

Plano (open-source, Katanemo) implements all four as a local proxy: one YAML config, zero agent code changes, plus a per-request cost console showing which model answered and what it cost. It reportedly cut Hermes agent usage 2x. Caveat: single agent, no controlled A/B, no task mix disclosed. Read it as an order-of-magnitude signal, not a benchmark.

Instrument the failure mode before adopting. Pinning locks a bad initial routing decision in for the task's lifetime, so you need post-commit quality monitoring and a runner-up fallback. The thing the 2x doesn't tell you is how often mid-task switches trigger cold re-bills on your workload. Most teams have zero visibility into that number, which is exactly why measuring it first is nearly free to justify.

The cost assumptions set at the late-2025 trough are now materially understated. Silicon Data's H100 one-year contract index rebounded from $1.70/hr to $2.35/hr, 38%, with spot up 10% YTD. Two independent series moving together is a credible demand signal. The thing this doesn't tell you is the index's tenor and region mix, which are undisclosed. Even granting that, a 38% swing flips build-vs-buy and reserved-vs-on-demand math that looked settled in Q4.

The memory leg compounds it. Micron's gross margins exceed their 2018 peak, memory stocks are up +200% to +700% YTD, and 2023 factory-investment shutdowns put new capacity years out. If you run HBM-backed GPU tiers, high-RAM serving nodes, in-memory feature stores, or managed vector DBs, this spike is already in your bill.

The contradiction worth acting on: hardware unit costs are rising while frontier API sticker prices fall. xAI's and Meta's new pricing is explicitly subsidized land-grabbing. That spread does not persist. Either API prices reprice upward when the traction phase ends, or entrants exit. Hard-wiring today's $1–2/M input pricing surviving 18 months is a structural risk, not a forecast.

The negotiator's tell sits in SpaceX's S1: a 3-year infrastructure deal with 90-day cancellation terms. In a genuinely runaway-demand market, providers don't hand buyers 90-day exits. Sophisticated buyers are paying today's prices while hedging supply relief. Structure the same way instead of locking rigid multi-year commits.

Rising per-hour costs also raise the payoff on right-sizing. A smaller model plus better retrieval that holds quality on your eval harness is worth more at $2.35/hr than at $1.70. The cheapest GPU-hour is the one you stop needing.