InfoKV, JetSpec, DeepSpec Stack for 4-9x vLLM Cost Cuts

Three Proof Points Converging

Open-weight models have claimed benchmark parity before. What's different this week is the simultaneous convergence of benchmark wins, cost data, and a named production deployment across multiple domains:

• Agentic tasks: Qwen-AgentWorld scores 73.8 on AgentWorldBench, against GPT-5.4 at 69.1 and Claude Opus 4.8 at 71.4 — Apache 2.0, with the eval harness included
• Coding: Ornith 1.0 claims 71.2% SWE-Bench Verified at roughly 1% of frontier inference cost, on a single RTX 4090
• Document extraction: Mistral OCR 4 reports 4-8 points over Azure, Google, and AWS, self-hostable in 2B/8B
• Production revenue: Coinbase cut AI spending about 50% while increasing token volume, by defaulting to open-weight models

The Pattern: MoE for Efficiency

The architecture is consistent: large total params, small active compute. Ornith is 30B total / 3B active. Qwen-AgentWorld is 70B / 12B active. You pay VRAM for the full parameter set, but throughput tracks the active slice. Capacity planning has to separate these two numbers. Conflate them and you either OOM or mis-estimate tokens/sec by a wide margin.

Where Sources Disagree

The Coinbase CEO claims about 50% savings. Multiple sources caution that the number likely bundles model switching with routing, caching, and batching changes that shipped alongside the migration. The clean attribution test holds routing constant and swaps only the model. That is the measurement that matters. Expect half the reported savings to come from the model switch and half from infrastructure work that would have helped regardless.

Benchmark scores are vendor-reported with no independent ablation. Qwen ships AgentWorldBench with the weights, which is the more valuable artifact: a reproducible agentic eval harness you control. The thing the score doesn't tell you is whether AgentWorldBench's construction has been audited. It hasn't. So the 4.7-point gap over GPT-5.4 is either a real edge or a tuned split, and the data here can't distinguish the two.

Open weights just made 'we need the frontier API' a claim you have to defend with your own eval harness, not assume. The eval harness is the asset, not the score.

The Cost Squeeze Context

This parity arrives as the closed-API cost curve bent upward. AWS +20% on inference, Google rationing Gemini capacity, GPT-5.6 gated behind U.S. government clearance. The supply side says prices go up and access gets constrained. The open-weight side says its models now match or beat on agentic benchmarks. Whether either claim survives contact with your production traffic is the open question. The cost pressure makes running the experiment mandatory rather than optional.

The Uncomfortable Convergence

Two independent research groups — a 12-system survey from SJTU/Tsinghua/MemTensor and MEMPROBE from UIC/KU Leuven/UCSD — published simultaneously and reached the same conclusion: no single agent memory architecture dominates. Effectiveness is entirely workload-dependent, and agents consistently fail at episodic retrieval and consolidation.

This matters because teams are still shopping for a 'best' memory framework. The literature now says stop shopping. Pick to workload. The research gives you the diagnostic to know whether your pick is working.

The Taxonomy That Replaces the Leaderboard

The multi-university survey formalizes agent memory into three levels:

1. Token-level: Short-term context window, manipulated by context engineering
2. Parametric: Learned weights, modified by fine-tuning or continued pretraining
3. Latent: Compressed state representations, the frontier research area

Functions map to factual, experiential, and working memory. The critical finding: what most teams call 'agent memory' is actually just RAG with extra steps, and the distinction matters because true episodic retrieval — recalling what happened during a previous task execution — is where agents measurably fail.

MEMPROBE: A CI-Ready Diagnostic

MEMPROBE tests memory via hidden-state reconstruction: can the agent reproduce information it processed earlier, without re-reading it? This is a ready-made metric you can lift directly into your agent eval harness. It distinguishes between an agent that 'remembers' by re-reading context and one that has actually consolidated information.

PydanticAI V2: Practical Context Compression

On the engineering side, PydanticAI V2's defer_loading=True targets the same bottleneck from a different angle. Multi-tool agents that stuff all tool schemas into context burn tokens and degrade routing accuracy simultaneously. Deferred loading keeps the prompt compact and loads workflows on demand. For any agent exposing 10+ tools, this is a near-free token reduction that compounds with the harness effects Raschka documented — where framework choice moves performance numbers as much as model choice.

The harness, not the model, decides whether the agent works. The ablation is the cheaper experiment, and it runs first.