GLM-5.2 Beats Opus 4.8 on Code at Half the Cost per Task

The Claim and Why It's Credible

GLM-5.2 is the first open-weight model where senior practitioners are flipping defaults, not writing thinkpieces. @_xjdr switched ncode's default to GLM-5.2. Cline's benchmarks show it producing cleaner output than Opus 4.8. It catches dead code and confirms production builds. Opus left type errors that passed the test suite. The API price is $1.40/$4.40 per million tokens. On real agent workloads that lands at $0.41/task versus $0.81 for Opus.

DeepSWE: 44% at $3.92/task average. Not frontier on every benchmark. Cheap enough to be the rational default for non-latency-critical agent loops. It ships on 20+ providers at >280 tok/s with <0.8s TTFT. That is production throughput, not a research demo.

The Multi-Model Stack Pattern

The homelab community is already running purpose-matched stacks. GLM-5.2 as planner at ~7 tok/s on 4x RTX 3090, slower but thorough. MiniMax 2.7 as coding executor at 45 tok/s, fits fully in VRAM. Qwen3.6 27B as checker/tester at 50 tok/s. This is Sakana's Fugu thesis, learned model routing, implemented by hand with full cost transparency.

Monolithic single-model agent loops leave performance and cost on the table. The question is no longer whether to use multiple models, but which role each model fills.

The self-hosting math: ~$6,000 for a 4x RTX 3090 + 192GB DDR5 rig. The community puts breakeven at ~5.5 years versus API pricing. On pure cost it does not pencil. Add availability guarantees, privacy requirements, and freedom from API rate limits, and it flips for high-utilization workloads. Especially while supply constraints keep Blackwell above $10/hour in bulk.

How to Run the Evaluation This Week

The headline cost ratio rarely survives a long tool-use trace. Cost-per-token is a vendor metric. Cost-per-successful-task is the only number that matters. Pick the three agent tasks burning the most Opus tokens this month. Replay them against GLM-5.2 with the same tools, system prompt, and retry policy. Log tokens, wall-clock, tool-call count, task success rate.

One caveat. Sakana's Fugu trails Opus on SWE-Bench Pro by ~10 points and the baselines are opaque. If the plan is a managed orchestration layer, demand cost accounting transparency before production. The DIY multi-model stack gives you that transparency by construction.

The Scale of the Problem

An audit ran 21 LLM judges across 9 providers through 541,000 judgments. Swapping exact-match agreement for Cohen's kappa, which corrects for chance agreement, deflates MT-Bench scores by 33 to 41 points. A 33-to-41 point gap is the difference between shipping a model and shelving it.

If your eval pipeline scores agreement with exact-match, it is overstating quality by 33 to 41 points relative to Cohen's kappa. That is not a margin you can paper over with a longer eval set.

The second finding is worse than the first: the ranking of which judges are reliable also reorders under kappa. The judge picked on agreement scores is not necessarily the judge that survives chance correction. Every downstream call that leaned on those scores, A/B tests, regression checks, model selection, is now suspect.

Why This Happens

Exact-match counts how often the judge agrees with the reference. If the judge always says "good" and 70% of examples are good, the judge gets 70% by doing nothing. Cohen's kappa subtracts expected chance agreement and exposes the judge's actual discriminative ability. Most LLM judges score high on agreement because they are biased toward positive evaluations, not because they are accurate. The spec called for discrimination; exact-match was measuring agreeableness.

What to Do

The fix is mechanical and the blast radius is not. Any model promoted, any regression signed off, any A/B test called on LLM-judge exact-match scores needs to be rerun under kappa. Teams that already switched report that the rank order of three candidate models inverted: the model that looked best was actually worst.

This lands directly on the GLM-5.2 evaluation from last week. Benchmark GLM-5.2 against Opus with an LLM judge scored by exact-match and the two models will look equivalent when one is meaningfully better. Score with kappa, or skip the judge entirely and use task success rate on real workloads as ground truth.