GLM-5.2 Beats Opus 4.8 on Coding at Half the Cost per Task

What happened

GLM-5.2 landed at #3 on GDPval-AA with 1524 Elo, behind Claude Fable 5 and Opus 4.8. The more interesting result is Cline's real-world bug-fix test, where GLM beat Opus 4.8. On the side-by-side, GLM confirmed the production build, removed dead code, and left no type errors. Opus left type errors that silently passed tests. Cost was $0.41 versus $0.81 per task.

This is the first open-weight win on an actual agentic coding loop. Not an isolated benchmark. A task routed through a real harness with tool calls, verification steps, and a production build check. GLM used more tool calls and heavier verification, which is the behavior you want from an autonomous agent and the behavior that static benchmarks tend not to measure.

Why the leaderboard number alone isn't enough

The same week, a 541K-judgment LLM-as-Judge audit showed that exact-match agreement, the metric most eval harnesses report, runs 33–41 Cohen's kappa points above the chance-corrected metric on MT-Bench. The implications:

• Quality gaps between models are systematically overstated by judge harnesses using exact match
• Judge rankings reorder under kappa correction
• Recent A/B tests gated on judge agreement may have been measuring noise

The model that just got cheaper is also the model that chance-inflated judges may have been quietly under-rating.

The deployment picture

Baseten, fresh off a $13B Series F and serving Cursor, Harvey, Notion, and Abridge, is the lead inference provider. The throughput numbers make latency parity with Opus plausible for most workloads. The thing this doesn't tell you is tail latency under burst load, which is what production agent loops actually hit.

The honest migration spreadsheet

The cost gap is real money. The honest calculation also includes retries from lower first-pass accuracy, longer chains of thought, engineering time to swap providers, and eval infrastructure upgrades needed to correctly measure the gap. If GLM holds half its bench-relative quality on internal evals, the migration pays off. If it holds a quarter, savings get eaten by retries.

Open-weight quality has reached the point where switching is a cost decision rather than a capability bet — but only for teams whose eval harness reports kappa instead of exact-match.

The study

A systematic audit covering 21 LLM judges across 9 providers and 541,000 judgments on MT-Bench puts numbers on something most eval teams have suspected for a while: exact-match agreement, which is the default in nearly every harness, overstates how reliable a judge actually is.

Cohen's kappa subtracts chance agreement from the headline figure. On the same judgment sets it runs 33 to 41 points below exact-match. On an imbalanced label distribution — which is what real model comparisons look like — two judges can agree 75% of the time and post a kappa of 35–42%. Most of that 'agreement' is class imbalance, not concordance.

What this breaks

• A/B shipping decisions gated on 'judge agrees Model A > Model B at 80%+ rate.' That 80% can correspond to a kappa of 40-47%.
• Model rankings on internal leaderboards. Judge rank order changes under kappa correction.
• Quality regression alerts. A 5-point exact-match drop may be noise. A 5-point kappa drop is signal.

If your harness scores on exact match, the reported quality gap between two models is almost certainly wider than the agreement-corrected gap. Eval harnesses built before this audit are probably overstating quality differences.

The practical fix

The correction is mechanically simple. The second-order consequences are not.

1. Add kappa computation to the existing judge pipeline. One function call using sklearn.metrics.cohen_kappa_score.
2. Re-score the last quarter of release decisions. The audit implies at least one ship/no-ship call flips under correction.
3. Expect tighter confidence intervals. If kappa shows models are closer than the exact-match score suggested, more eval samples are needed to reach the same statistical power for model selection.

Connection to GLM-5.2 evaluation

This audit directly affects how GLM-5.2 should be evaluated against Opus. If the judge reports GLM trailing Opus by 8 exact-match points, the kappa-corrected gap is likely 3–4 points. That is well inside the range where a 50% cost reduction pays for the switch. The eval methodology determines whether the migration looks viable.

Distinct from prior coverage: earlier this week's eval harness note covered mode collapse and diversity metrics. Today's issue is orthogonal. The agreement metric itself is inflated by chance, which affects every judge-based comparison regardless of what the judge is scoring.