The harness is the product now — models stopped improving at agent tasks

Princeton's updated ICML 2026 paper — Towards a Science of AI Agent Reliability — added GPT 5.5, Gemini 3.1 Pro, Gemini 3.5 Flash, and Claude Opus 4.7 to its framework and came back with the same answer: no meaningful improvement over the models they replaced. The ALE benchmark, which maps 1,000+ tasks to the U.S. occupational taxonomy, shows a 2.6% full-pass rate at the hard tier. SWE-Marathon tests coherence across 1 billion tokens — Slack-clone builds, JAX-to-PyTorch rewrites — and agents fall apart well before the budget is exhausted. Three independent labs, different objectives, different alignment stacks, converging on the same ceiling. When that happens, the constraint is in the problem, not the lab.

The practical consequence: if your roadmap sequences features behind "ship when the model gets more reliable," that sequencing has no exit condition. The waiting strategy just lost its premise.

Yes, but — one paper isn't a ceiling, it's a measurement, and a genuine reliability step-function could arrive from any of the three labs inside two quarters. That's worth holding. What makes the take still stand is that three generations have now passed without that step-function materializing on multi-step agent tasks — and the open-weight floor is rising fast enough that the closed-model reliability premium is compressing regardless.

The harness, not the model

GitHub's CPO disclosed that March 2026 produced 17 million agent-authored pull requests — a 3x overshoot of internal capacity forecasts, traced to the December 2025 background-agent GA. That volume saturated West Coast infrastructure and forced emergency Azure load-shedding. The failure mode wasn't model quality. It was retry behavior: an agent that doesn't see the queue opens another PR to fix the timeout. The harness around the model either handles that gracefully or compounds it.

One concrete result from the Princeton work worth internalizing: hand-rolled raw API calls used 6x more tokens at lower success rates than purpose-built CLI tooling. Tool abstractions that encode domain knowledge, validation, and workflow shape are a measured reliability and cost lever — not a convenience. This is the lever most teams haven't pulled because they're waiting for a smarter model to make it unnecessary. It won't.

Copilot's June 1 shift to usage-based billing makes this a billing line item, not just a quality concern. Under seat pricing, a semantic router was a nice-to-have. Under usage pricing, the cost delta between routing 60% of traffic to a Flash-class model versus sending everything to a frontier model is roughly $1M per $10M spend — Cloudflare AI Gateway's per-model spend caps with automatic fallback validated exactly that number in production. Single-model-everywhere no longer survives a usage-based invoice.

The supply chain is actively compromised

While the reliability story is about engineering discipline, the supply chain story is about an incident happening right now.

The Miasma worm compromised 73 Microsoft GitHub repositories across four organizations and remains uncontained. Its companion campaign, IronWorm, poisoned 50+ npm packages with Rust-based info-stealers — compiled native binaries that npm audit cannot see, harvesting CI tokens, SSH keys, and .npmrc credentials to propagate further downstream. This is not a manual typosquat campaign. It's self-replicating. Every CI run that pulls a contaminated package becomes a new propagation node.

At the same time, a critical RCE in Hugging Face Transformers — 2.2 billion installs — fires not on weight deserialization but on model config loading. The trigger path is trust_remote_code=True in from_pretrained() calls, where config.json / auto_map fields reference and execute arbitrary code. Research workstations evaluating candidate models against the Hub are the primary target: a data scientist pulling ten repos in a sitting touches ten potential exploit paths on a machine with cached cloud credentials.

An AI agent from an unnamed security startup found 21 zero-days in FFmpeg in a single research cycle. FFmpeg sits underneath torchvision.io, decord, PyAV, OpenCV, and Whisper audio preprocessing. If a data loader runs read_video() on scraped content in the same process as the trainer, a malicious MP4 has a path to wandb keys, S3 credentials, and model checkpoints.

CVE-2026-20245, a CVSS 7.8 flaw in Cisco Catalyst SD-WAN Manager, is under active exploitation with no patch available. When the sole network provider has no fix, compensating controls are the only option.

OpenAI's response to the prompt injection problem is instructive: Lockdown Mode disables Deep Research, Agent Mode, and web image fetch entirely. Not hardened — disabled. When the best-resourced lab in the space ships its fix as a kill switch, the signal is that model-layer defenses have unacceptable false-negative rates against a determined attacker. Every agent you're building touches the same primitives.

What to do this week

These aren't parallel stories. They're the same story: the model layer isn't where the leverage is, and the infrastructure underneath it is under active attack. The teams that move this sprint on both fronts are the ones operating on accurate premises.

On reliability: add consistency@k (N≥5 trajectories) as a first-class metric alongside pass@1 in your eval harness, and audit your eval scaffolds for answer leakage — check what the agent can see during eval that it shouldn't. The GAIA-level benchmarks had this bug; internal leaderboards likely do too. Then instrument cost-per-merged-PR and tokens-per-resolved-task before the June billing cycle surfaces the numbers for you.

On supply chain: run npm ls against published Miasma/IronWorm IOC lists across all repos today, looking specifically for postinstall scripts downloading native binaries. Rotate npm publish tokens, GitHub PATs, and CI runner credentials for any developer who installed npm packages in the last 30 days. Pin Hugging Face Transformers to the patched version, set trust_remote_code=False in CI by default, and grep every from_pretrained() call in the codebase. For Cisco SD-WAN, restrict the management interface to jump-host-only ACL now — that's the only available compensating control.

The deeper architecture decision — agents that read untrusted content should never hold write access to privileged state without an out-of-band authorization step that can't be bypassed through conversation — doesn't require waiting. Meta's chatbot was socially engineered into changing Instagram account emails. No technical exploit. Just a model with too much scope and no deterministic policy layer underneath it. If a policy can be argued out of its decision in English, it's not a policy. It's another model.