GPT-5.4 ships above the human bar — and breaks past 256K tokens

GPT-5.4 landed this week with the kind of benchmark line that gets quoted in board decks: 75% on OSWorld-Verified, above the 72.4% human baseline, doubling GPT-5.2 in a single generation. 83% win-or-tie against professionals across 44 GDPval categories. APEX-Agents went from sub-5% to over 50% in twelve months. Token use down 47%. Input price $2.50 per million — half of Opus.

That's the press release. The fine print, published by OpenAI itself, is more interesting and considerably more useful.

The 256K ceiling nobody is putting on the slide

OpenAI's MRCR v2 long-context benchmark — their own numbers, not a third-party teardown — shows accuracy falling off a cliff. 97% at 16-32K tokens. 57% at 256-512K. 36% at 512K to 1M. Worse than a coin flip on retrieval tasks once you cross the halfway mark of the advertised window.

This is not a GPT-5.4 problem. It's a transformer problem, and every frontier model with a million-token window has a version of it. But GPT-5.4 is the one being marketed hardest right now, and the gap between what the spec sheet implies and what production behavior delivers is wide enough to ship broken software through.

If your pipeline assumes faithful retrieval over a full codebase, a quarter of legal discovery, or a quarter of Slack history dumped into a single prompt — you are shipping a system that silently fails on the majority of long-context queries, and unless you've instrumented needle-in-haystack probes at your actual operating lengths, you don't know which queries.

The cliff is not gradual. It's around 256K. That's your reliability ceiling. Treat anything beyond it as a separate architecture problem — chunking, compression, recursive subagents, KV-cache compaction. Baseten's Attention Matching work shows 65-80% retention at 2-5x compression, which beats raw context past the cliff. Cursor's cloud agents already solve this at the systems layer: subagents whose output gets summarized before it reaches the parent. The bigger window isn't the answer. The architecture around the window is.

The benchmark numbers were run at $80-per-prompt mode

Every headline GPT-5.4 score was produced at "xhigh" reasoning effort — a setting where a one-word "Hi" prompt costs $80 and takes five minutes. The model the rest of us will actually call, at standard or medium reasoning, has no published benchmarks at all.

That's a real gap. The 47% token efficiency claim is self-reported. The 33% hallucination reduction over GPT-5.2 ships without methodology. Multiple users are already reporting that GPT-5.4's deliberate conversational "loosening" has introduced prompt leakage, unrequested feature additions, and hallucinations in structured output flows — a reliability regression that won't show up on any leaderboard but will show up in your support tickets.

Benchmark on your prompts, at your reasoning effort, against your acceptance criteria. Not OpenAI's.

The cost story is the one worth acting on

While the headline benchmarks are noisy, the pricing signal isn't. GPT-5.4 at $2.50/M input is half of Opus. DeepSeek V4 is imminent — trillion parameters, 32B active MoE, trained entirely on Huawei and Cambricon silicon, and the early numbers claim financial document classification at $210/month versus $4,200 on GPT-5 with accuracy within two points. Anthropic's diversified compute story produces 30-60% lower cost per token than OpenAI's Nvidia-anchored stack. Ridgeline is doing 73-88% on SWE-bench at $29/month for 100 PRs.

You can argue with any single number on that list. The direction is unambiguous: the frontier is paying a 19x premium for roughly 0.6 percentage points of benchmark improvement, and that gap is closing every quarter.

The rational architecture is no longer single-model. It's task-routed. Commodity classification and extraction goes to the cheapest open-weight model that clears your quality bar. Fast cheap general inference goes to a Flash-tier or GPT-5.3 Instant. Complex reasoning goes to Thinking. Agentic tool use goes to GPT-5.4 standard. The premium tiers are barbells now — near-free at one end, $80-per-prompt at the other, and the middle is getting squeezed.

If you don't have a model-agnostic gateway in front of your inference calls, you're carrying technical debt that compounds with every new model release. Ten frontier models shipped in twenty-eight days. Manual quarterly evaluation cannot keep pace.

And the security model hasn't caught up

February 17: an attacker put a prompt injection in a GitHub issue title. An AI triage bot read the title, interpreted it as an instruction, leaked an npm publish token, and a byte-identical Cline package with one extra line backdoored roughly 4,000 developer machines. Same week, Perplexity's Comet exfiltrated local files via a calendar invite. Same class of bug in Chrome's Gemini panel.

99% of dev teams use AI code assistants. 29% have formal AI security controls. The gap is the attack surface.

The Cline chain is the template every agent in your SDLC is now exposed to: untrusted text input, LLM with tool access, credential exfiltration, supply chain poisoning. Prompt-level guardrails operate at the same abstraction layer as the attack — they cannot be the answer. The fix is system-level: hard allowlists on what resources an agent can touch, least-privilege credential scoping, sandboxed execution. Perplexity's Comet patch was a hard file:// block, not a better system prompt. That's the right architectural move.

What to do this week

One action, specific. Pick your three highest-volume LLM endpoints, instrument MRCR-style needle-in-haystack probes at your actual operating context lengths, and publish the accuracy curve on an internal dashboard by Friday. Anything past 256K with under 70% retrieval accuracy gets a chunking-and-retrieval fallback in the next sprint, full stop.

The model got better. The pricing got better. The architecture you wrap around it is what will determine whether any of that translates into production reliability. The 1M-token window is a marketing number. The 256K cliff is your engineering one.