Stanford: 80% Accurate LLM Router Cuts Costs 59%, Energy 64%

Stanford Proved the Economics; Three Model Releases Made It Practical

Stanford's study across 20+ models on 8 accelerators with over 1M single-turn queries delivers a clear engineering spec: build a router that classifies query complexity, send easy queries locally, send hard ones to cloud frontier models. The router only needs 80% accuracy to deliver 59% cost reduction, 61.8% compute reduction, and 64.3% energy reduction against batched cloud baselines.

The reason this works: most production LLM traffic is easy. Knowledge retrieval, text formatting, simple summarization. Local model coverage of ChatGPT-equivalent queries grew from 23.2% in 2023 to 71.3% in 2025, with a 5.3x gain in intelligence-per-watt. Extrapolating this trajectory, 85%+ of queries become local-serviceable within 12-18 months.

The router is the cheapest infrastructure you can build. It only has to be right four times in five.

The Local Tier Just Got a Frontier-Class Option

LongCat-2.0 (MIT-licensed, 1.6T total parameters, ~48B active via MoE) scored 59.5% on SWE-bench Pro versus GPT-5.5's 58.6%. At 48B active parameters, inference requires roughly 2x H100s for active computation — expensive but feasible for a single team. The 1M native context window was trained, not interpolated, meaning attention patterns over long codebases are learned rather than approximated. Caveat: benchmarks are self-reported by Meituan, though 'Owl Alpha' topping OpenRouter charts for 2 months provides some independent signal.

For lighter workloads, Qwen 3.6 27B achieves 32 tokens/sec on an M5 MacBook via llama.cpp — viable for code documentation, boilerplate generation, and log analysis with zero cost-per-token.

The Critical Design Constraint

Coverage is strong on chat and knowledge tasks but weak on hard technical reasoning. A router keying on surface features (prompt length, keyword density) will route hard problems to the wrong tier. Classify on reasoning complexity, not superficial indicators. The Stanford paper designates this as the F29 failure category — the queries where local models confidently produce wrong answers.

Serving Framework Matters More Than You Think

On AMD MI355X, sglang outperformed vLLM and ATOM serving GLM-5.2, hitting 2626 tok/s/node. The path was configuration, not custom kernels: MXFP4 quantization via AMD Quark, FP8 KV cache, and tuned MoE kernel selection. This delivered 2x better cost-per-token than Blackwell at the serving level — the number that appears on your bill. If you standardized on vLLM 12 months ago, that was reasonable then. Validate it against sglang on your actual workloads before trusting it again.

CNI and cgroups v1 Are Gone, Not Deprecated

Podman 6.0 removes CNI networking and cgroups v1 support. Read that as removal, not a deprecation warning. Hosts on RHEL 7-era kernels will not boot Podman 6.0. CI infrastructure that hasn't explicitly enabled cgroups v2 will fail the same way. Podman 6.0 will not work in either case. The netavark/Pasta migration is documented. If you wrote custom CNI plugins, budget real time. The docs don't make that part painless.

The AMD GPU passthrough addition is the tell. Red Hat is positioning Podman for ML inference workloads, going after Docker's GPU runtime for local model serving. That connects to the hybrid inference architecture above. The local model serving tier may end up on Podman with GPU passthrough instead of Docker.

Kepler Drops eBPF: Counter-Narrative to eBPF Maximalism

Kepler's Kubernetes power-monitoring rewrite drops eBPF entirely. It uses RAPL counters and dynamic hardware discovery instead. The mechanism is the argument: eBPF needs elevated privileges, and security-hardened clusters won't grant them. The rewrite trades per-process energy attribution for deployment that works everywhere without privilege escalation. If you're working toward EU sustainability reporting mandates, this rewrite makes Kepler actually deployable in production.

Microsoft WSL Containers: Docker Desktop Competition

Microsoft's WSL container preview runs on virtiofs for approximately 2x file I/O improvement and ships Defender and Intune integration. On Windows dev environments this is the first serious challenge to Docker Desktop. Docker's own licensing changes did the recruiting. Enterprise teams already had one foot out the door.

The standalone incident management category is being absorbed into the observability layer, exactly the way standalone log management got absorbed a decade ago.