AIinferenceeconomics

In April 2026, frontier API prices doubled while Kimi K3’s open weights landed on July 27 matching GPT-5.6 on coding benchmarks at roughly 40% lower cost, self-hosting cleanly on vLLM. Those two data points, taken together, invert the assumption most inference stacks were built on: that hosted frontier tokens would keep getting cheaper and open weights would stay a generation behind. Neither is true anymore. The serving-dollar map has been redrawn, and most engineering orgs are still spending against the old one.

The Cost Equation Just Inverted

For three years, the default architecture was straightforward: route everything to a frontier API, absorb the price cuts as they came, treat open weights as a research curiosity. That reasoning is now broken in both directions.

On the hosted side, prices aren’t just failing to fall — they’re rising in ways dashboards can’t see. Sonnet 5’s tokenizer change amounted to a 42% stealth price hike with no movement on the price card. Frontier providers, having burned through the early land-grab phase, are now optimizing for margin. The published rate is a decoy; the effective rate is set by tokenization, context handling, and reasoning-token overhead that shows up only in the invoice.

On the open side, the ceiling collapsed. Thinking Machines’ Inkling sits at 41 on the Intelligence Index under Apache 2.0 while cutting output tokens 40% at equal-or-better quality. Kimi K3 edges Opus 4.8 on coding. The self-host path is no longer a quality compromise — it’s a cost decision with a quality bonus on specific workloads. Microsoft’s decision to replace OpenAI inside Excel and Outlook with its own models is the enterprise-scale version of the same math: when parity is available, the $13B relationship doesn’t hold.

The practical consequence is that any stack still routing 100% of traffic to a single frontier vendor is overpaying by a factor that grows every quarter. And the overpayment is compounding on top of a second, structural mistake.

Agents Are CPU Workloads Wearing GPU Costumes

An agent trace, decomposed honestly, is 70–80% orchestration: tool calls, JSON parsing, retries, API roundtrips, filesystem reads, database lookups. The actual token generation — the part that needs a GPU — is a minority of wall-clock time and an even smaller share of compute. Yet the industry standard is to pin the entire agent loop to a GPU instance, paying H100 rates for what is mostly network I/O and Python.

The overspend runs 2–4x on agent-heavy workloads. And the workloads are getting more agent-heavy, not less. Airbnb now auto-resolves over 40% of guest support cases with zero human agents — the shape of that traffic is dozens of small tool-calling turns per resolution, each one paying GPU tax on CPU work.

This is why Meta’s multi-billion-dollar Graviton5 order matters more than it looks. It’s not a diversification hedge against NVIDIA supply — it’s an acknowledgment that the agent tier of inference belongs on ARM cores at a fraction of the per-hour rate. AWS Graviton instances handle tool-calling orchestration, embedding lookups, retrieval, and result parsing at CPU prices; the GPU only wakes up for the generation step. Splitting the workload this way is the single largest architectural cost lever available to any team running agents at scale, and it’s the one Meta is buying millions of cores to exploit.

Expect the segmentation to sharpen. GPU capacity gets reserved for training and for the generation tokens of high-throughput chat. Everything else — the orchestration layer, the RAG pipeline, the eval harness, the guardrail models — migrates to Graviton or equivalent ARM silicon. The teams that make this split are running the same workloads their competitors are running on H100 fleets, at 30–50% of the cost.

Prompt Caching Is the Free Lunch Nobody Ate

Before any hardware conversation, there is a software optimization sitting in plain sight that most teams still haven’t implemented. Prompt caching — reusing the KV-cache state of a stable system prompt or long context across requests — cuts costs 50–90% on chat-shaped workloads. The math is trivial: if 8,000 tokens of system prompt and tool schema are identical across every request, paying to reprocess them on every turn is pure waste.

The reason it hasn’t been adopted isn’t technical difficulty. Every major API supports it. The reason is that it requires deliberate prompt architecture — stable prefixes, careful ordering, cache-aware batching — and most codebases evolved organically, with system prompts assembled dynamically from a dozen sources. Refactoring for cache hits is unglamorous work that pays back immediately.

Combine caching with the CPU/GPU split and the compounding is significant. A team paying $100K/month on a frontier API for an agent product can, without changing model quality, typically land at $25–40K by caching aggressively, moving orchestration to Graviton, and routing non-critical calls to a self-hosted open-weight model. Nothing in that path requires research; it requires an engineer with a month and a mandate.

The Bill Is Now Visible on the P&L

The capital side of this equation used to be abstract. It isn’t anymore. Alphabet’s EPS fell 7.7% despite 18.5% revenue growth — infrastructure spend is now compressing margins on the largest, best-run AI business in the world. That’s the signal for everyone downstream. The hyperscalers can no longer subsidize inference indefinitely; the cost gets passed through in tokenizer changes, in rate limit shifts, in the quiet retirement of cheaper model tiers.

At the same time, AI chip rounds have reflated 4–8x, with SambaNova marking from $2B to $11B, mostly on companies with no material revenue. Nvidia’s $20B Groq deal set a strategic-M&A ceiling that private silicon rounds now use as a floor. The capital is betting that inference demand will absorb any amount of specialized silicon that ships. That may prove correct in aggregate, but for individual buyers it means the vendor landscape will churn violently over the next 18 months, with pricing that reflects fundraise pressure more than unit economics.

The implication for anyone building on top: assume that hosted prices trend up, not down, for the next four quarters. Assume that model-parity events like Kimi K3 will keep arriving, roughly one per quarter, from labs whose names weren’t on the map last year. Assume that the winners are the teams whose serving architecture can absorb a model swap in a week, not a quarter.

Operational Posture

For this quarter, four moves matter:

1. Audit tokenizer-level costs, not price-card costs. Pull the last 30 days of actual invoices and compute effective per-request cost. If it drifted upward without a stated price change, that is the tokenizer tax and it will keep compounding. Build the dashboard around invoice reality, not vendor-published rates.

2. Split the agent loop across CPU and GPU tiers. Move tool-calling orchestration, retrieval, parsing, and guardrail models to Graviton or equivalent ARM instances. Keep GPU capacity for generation only. Target a 2–3x reduction in per-agent-turn compute cost.

3. Implement prompt caching before touching model selection. Refactor system prompts and tool schemas into stable, cache-aligned prefixes. Instrument cache hit rates. Chat-shaped workloads should hit 50%+ savings inside two sprints.

4. Stand up one open-weight model in production behind a router. Kimi K3, Inkling, or equivalent on vLLM. Route a defined slice of traffic — internal tools, low-stakes classification, high-volume summarization — to it. The point isn’t to displace the frontier vendor immediately; it’s to have the muscle in place when the next parity event arrives, because it will.