Cost-Per-Correct-Answer:TheEvalMetricFinanceWillForce

◆ DEEP DIVES

1. 01

   The Harness Quarter: Token Efficiency, Eval Economics, and the $500/Day Developer

   The New Axis Your Eval Harness Is Missing

   Accuracy-only model evaluation is now actively misleading for production decisions, and this week's evidence is specific enough to act on. The invoice is where model choice gets decided this quarter, not the leaderboard.

   IBM's Granite 4.1 8B spent 4M output tokens on the Artificial Analysis Intelligence Index against 78M for Qwen3.5 9B, a 19.5× efficiency gap at comparable capability. That gap does not appear on any accuracy leaderboard. It appears on the invoice. Factory AI's 13-model code-review bakeoff found a $1.25/PR model beat one costing more than 2× on real pull requests, with token spend uncorrelated to review quality on their harness. Harness engineering, iterating on prompts, tools, and middleware with the model held constant, lifted Terminal-Bench 2 pass@1 from 69.7% to 77.0% in 10 iterations, beating the human-designed Codex-CLI baseline at 71.9%.

   The harness, not the model, is where this quarter's agent economics live.

   ---

   The Cost Crisis Is Already Here

   Across 15 companies surveyed by The Pragmatic Engineer, AI coding token spend has grown 10–15× in six months. Individual developer burn is hitting $500/day, and one developer burned $10K in a week because of a caching bug. A 2,000-person SaaS cut costs 30% by changing the default model from Opus to Sonnet, which is a config change rather than an architecture change.

   On the eval side, evaluation compute has quietly crossed from overhead to bottleneck. Individual eval runs now cost tens of thousands of dollars. DeepMind's ProEval pattern, surrogate models scoring checkpoints cheaply with full evals reserved for release candidates, is the direction of travel. Netflix published the most portable eval blueprint of the quarter: ~600 expert-labeled golden examples with tiered reasoning and consensus scoring, landing at 83–92% accuracy across four quality dimensions.

   The awkward finding: a community benchmark showed the two-word prompt "be brief" matched a purpose-built compression plugin on both token reduction and output quality. Trivial baselines embarrass sophisticated prompt frameworks more often than practitioners admit. The thing this doesn't tell you is whether the plugin still wins on tasks the benchmark didn't cover, which is worth checking before ripping it out.

   Optimization Lever	Reported Impact	Effort
   Opus→Sonnet default swap	30% cost cut	Config change
   Harness engineering (10 iters)	+7.3pp accuracy, no model change	1-2 sprints
   Token-efficient model swap (Granite)	19.5× fewer tokens	Eval + migration
   Surrogate eval (ProEval pattern)	50%+ eval compute savings	Quarter-long build
   "be brief" vs. plugin	Parity on token reduction	Two words

   ---

   What This Changes

   The binding constraint has shifted. Eval harnesses that report accuracy without token cost are no longer decision-grade instruments. Every model candidate should produce (accuracy, tokens-consumed) pairs plotted on a Pareto frontier. The Granite vs. Qwen gap only surfaces when token counts are logged per task. The Factory AI finding only surfaces when cost is a first-class eval dimension.

   Microsoft's cloud margin dropped 5 percentage points to 56%, attributed explicitly to inference-heavy AI workloads including GitHub Copilot. Consumption pricing makes token-per-request a product metric rather than an infra footnote. Teams without per-request token accounting will learn their cost structure through a Finance escalation, not a dashboard.

   Action items

   • Add $/correct-answer and tokens-per-task as first-class metrics in your model eval harness alongside accuracy
   • Run one iteration of agentic harness engineering on your top agent benchmark, holding the model constant
   • Build a 600-example golden eval set with tiered + consensus LLM-as-Judge scoring for your highest-stakes generative output
   • Implement surrogate-model eval (ProEval pattern) for continuous checkpoint scoring, reserving full benchmark runs for release candidates

   Sources:AINews · The Pragmatic Engineer · TLDR AI · TLDR Dev · ben's bites · Aaron Holmes

2. 02

   Your ML Serving Stack Is Being Exploited Faster Than You Can Patch

   Disclosure-to-Exploit Has Collapsed to Hours

   One seven-day window is a small sample, but the signal is hard to ignore. LiteLLM's pre-auth SQL injection (CVE-2026-42208), which fronts most teams' OpenAI, Anthropic, and Bedrock keys, was exploited within 36 hours of disclosure. LMDeploy's SSRF went from advisory to working exploit in 12.5 hours, and no public PoC was available. The plausible mechanism is attackers feeding CVE advisories into LLMs and getting usable exploits back.

   Patch-window SLAs were calibrated for a world where writing the exploit was the slow step. On this data, it is not the slow step anymore.

   ---

   Pickle Deserialization Is the 2026 Pattern-of-the-Year

   Three independent ML frameworks shipped the same bug class this week, unsafe deserialization of attacker-controlled artifacts, each at CVSS 9.8:

   Framework	CVE	CVSS	Attack Surface
   LeRobot ≤0.5.1	CVE-2026-25874	9.8	HF's robotics stack; checkpoints shared broadly
   KTransformers ≤0.5.3	CVE-2026-26210	9.8	Local inference on GPU hosts with credentials
   Pipecat	CVE-2025-62373	9.8	Voice/multimodal pipelines, network-facing

   On top of that: Claude Code's CVSS 10.0 sandbox escape via symlinks (CVE-2026-39861, patched in v2.1.64), NVFlare Dashboard's auth bypass (CVE-2026-24178, CVSS 9.8) in federated learning deployments, and 73 GlassWorm-linked fake extensions on Open VSX in April alone. That is the marketplace behind Cursor, VSCodium, and most VS Code forks.

   ---

   The Supply Chain Cascade Is the Structural Threat

   The TeamPCP/UNC6780 chain is worth tracing end to end. They poisoned checkmarx/kics:latest on Docker Hub. Bitwarden's Dependabot automation pulled it into CI. The malicious code shipped as @bitwarden/cli 2026.4.0. That is the exact automation topology most modern ML teams run: tag-based image pulls, bot-driven dependency updates, CI runners holding model registry credentials.

   In parallel, DPRK's HexagonalRodent is targeting ML and Web3 engineers through fake-recruiter coding assessments that abuse VSCode tasks.json auto-run. The scorecard so far: $12M exfiltrated across 2,726 developer machines in Q1 2026.

   LiteLLM deserves separate attention. Running it as a cost and routing gateway funnels every upstream provider key into a single database that sits behind a pre-auth endpoint. The working assumption should be that any keys transiting the service during the vulnerability window are burn-worthy. Rotate first, investigate second.

   ---

   Two Lever Fixes Cover Most of the Surface

   One policy change removes most of this week's blast radius: ban tag-based image pulls and pickle-loads-from-untrusted-sources in CI. Both are enforceable with a linter and a registry policy, and between them they would have blocked the majority of the ML-relevant exposure listed above. The thing this doesn't cover is the pre-auth SQL and SSRF class. That one is still a patch-speed problem.

   Action items

   • Rotate all provider keys (OpenAI, Anthropic, Bedrock) that transited LiteLLM and pull 30-day usage anomaly reports from each provider
   • Enforce torch.load(weights_only=True) or safetensors as a lint rule and block pickle artifacts from untrusted sources at the artifact-store layer
   • Pin all ML base images by sha256 digest, disable Dependabot auto-merge for container and pip dependencies, and set a <24h patch SLA for ML serving CVEs with technical detail
   • Disable VSCode 'folderOpen' auto-tasks org-wide and require external repos to run in disposable devcontainers with egress allowlisting

   Sources:SANS AtRisk · TLDR InfoSec · CSO Update · CSO First Look · Daniel Miessler · TLDR IT

3. 03

   Agent Governance Has Legal Teeth Now — And Your Metrics Aren't Measuring the Right Thing

   The 93% Problem

   Anthropic disclosed that 93% of AI agent actions are auto-approved by human reviewers. At that rate, the approval signal is not a safety control. It is a gauge that happens to have a human attached. The signal carries near-zero mutual information with risk. An agent that only ever takes the same three actions in risky contexts is not safe. It is under-explored, and this metric cannot tell the difference.

   Anthropic's own recommendation is to move from per-action approval to continuous policy monitoring. The metrics that carry actual signal are override rate on high-risk actions, policy-violation count per 1,000 traces, and action entropy per agent. A new agent version that pushes auto-approval from 93% to 96% is not an improvement. It is the same broken gauge reading slightly higher.

   An agent eval harness without override rate, cost-per-task, and tool-boundary violations is measuring demos, not agents.

   ---

   Courts Just Made Autonomy a Design Parameter

   A Northern District of California court ruled under Rule 10b-5 that when a platform's AI exercises 'ultimate authority' over assembled content, the platform is a 'maker of fraudulent statements'. This is the first U.S. ruling to reject the premise that autonomous AI output is legally equivalent to user-generated content. Section 230-style shields do not automatically extend to agent decisions.

   Autonomy Level	Control Flow	Liability Posture	Engineering Signal to Log
   Tool-assist	Human → AI suggests → human commits	Shield likely holds	suggestion_id, accept/reject
   Human-gated agent	AI drafts → human approves → commit	Shield likely holds	approval_latency, reviewer_id, diff
   Autonomous agent	AI assembles and commits unreviewed	Platform exposed as 'maker'	Full chain, model version, guardrails

   Most production agentic stacks today ship in the third row and log like the first. The thing this gap doesn't tell you, until discovery does, is whether a human actually closed the loop. Observability stacks capture latency, token spend, and output quality. They rarely capture a first-class authority_level field attesting to human approval.

   ---

   Offensive Capability Is Outpacing Defensive Measurement

   MOAK's agent built working exploits for 174 of 178 KEVs published after model training cutoffs, using publicly available Opus 4.6 and GPT-5.4 behind ordinary scaffolding. XBOW reports GPT-5.5 cut its miss rate from 40% to 10%, with black-box performance now exceeding what GPT-5 managed with source access. Persistence-on-failing-paths halved. That is a planning improvement, and planning improvements transfer to any long-horizon agent task.

   HackerOne paused its Internet Bug Bounty because AI-driven submission volume is outpacing remediation bandwidth. The same queue-overflow dynamic will replay inside every org: PR review, model-card approvals, security alerts. Any human-review queue in the ML stack is one capability jump from the same asymmetry.

   Action items

   • Add authority_level (tool_assist | human_gated | autonomous) and human_reviewer_id to every agentic decision record, persisted for artifact lifetime
   • Replace per-action approval metrics with a continuous-policy dashboard tracking override rate, policy-violation rate, and action entropy per agent
   • Gate destructive operations (DB writes, deploys, credential access) behind dry-run + human approval regardless of model confidence
   • Run FinBot CTF (OWASP) against your agent stack as a pre-production red-team gate, specifically testing MCP tool-description tampering and cross-tenant context bleed

   Sources:TLDR IT · Future Perfect · Clint Gibler · CyberScoop