OpenAI Eval Model RCEs Hugging Face Prod via Registry Proxy

Per-token price is not cost. Cursor rebuilt SQLite from spec and got identical passing output for anywhere between $1,339 and $10,565. Same model, same result. The 8x spread came entirely from the orchestration config. Kimi K3 looks cheap at roughly half GPT-5.6-Sol's per-token rate, then burns about 2x the tokens, which nets out to a wash. It also ships max-effort-only reasoning, so a single icon cost testers 13,000+ reasoning tokens. Benchmark cost-per-completed-task on real traces from the actual workload. The leaderboard rank will mislead otherwise.

Latency is now a hard number, not a vibe. NiCE's field data puts Opus 4.7 above 4 seconds to first token. Gemini 3 Flash sits around ~1s, and users tolerate about 2.5s before they decide the bot is broken. So Opus is disqualified for voice, and not on intelligence. Its error rate is genuinely half Gemini's. That gap is a routing policy, not a model choice: latency-sensitive turns go to Gemini 3 Flash or the mini tiers, accuracy-critical async work goes to Opus.

The third axis is security posture, and buyers keep skipping it. Sonnet 5 ships a deliberate cyber-capability cap. Grok 4.5 undercuts on price and carries the thinnest safety documentation of the set. GPT-5.6 reportedly sits at cyber-vulnerability parity with an export-controlled model. There is a pattern here: the cheapest and most capable-looking option keeps turning out to be the least documented one.

Ramp Router is the working example of doing this right. EWMA tracks recency-weighted provider failure rates. Thompson sampling runs over latency. Together they make a picker that is fault-aware and deadline-aware, and it cut inference cost 30% with no quality loss. If building something similar, start with the EWMA failure tracking and defer the bandit layer. That ordering is deliberate. The failure tracker is most of the leverage for a fraction of the effort.

Here is the mechanism most quantization writeups skip: past roughly 6.7B parameters, transformer layers develop hidden dimensions with values 20-100x larger than everything else. Call it 0.1% of features. They carry a disproportionate share of text prediction. If those outliers inflate the quantization scale factor, the model breaks. Bit width is not the variable that matters. Outlier handling is.

The methods differ by where they defend those outliers. RTN doesn't defend them at all, which is why it should never run below 8-bit in production. GPTQ compensates post-hoc using calibration data. It is fast, but the fit is tight to the calibration set, and when production traffic drifts, degradation shows up quietly. AWQ takes the other route: it pre-scales the ~1% of channels that get multiplied by large activations before rounding. That targets structural importance, not calibration statistics. The output is plain INT4 with no special-case inference logic, which is the actual reason vLLM ships it as the default. LLM.int8() routes outliers through a separate FP16 path. Correct, but slow. Fine for local dev and QLoRA. Not fine for serving.

The infra math is where this pays off. A 70B model at FP16 needs 140GB. An H100 has 80GB. So before the first token, the bill includes NCCL overhead, cross-node comms, and a multi-GPU blast radius. At 4-bit the same model is ~35GB and fits a single 40-48GB card. Any 65B+ model currently sharded across GPUs is worth an audit: count the orchestration that disappears on AWQ int4. In most cases it is most of it.

One caveat from the accuracy side. If a fine-tuning pipeline is hitting a ceiling on post-hoc methods, Gemma 3's QAT recipe is worth a look. It distills from the full-precision model in about 5,000 steps. That is cheap enough to justify a sprint, and cheap enough to abandon if it doesn't move the number.