Four Studies Converge: AI Productivity Gains Overstated by Users

The Convergence

Four independent studies this cycle — 5,179 support agents, a 16-developer RCT, ~758 BCG consultants, and 300 enterprise deployments — converged on one read. AI productivity gains are real, heterogeneous, jagged, and systematically over-reported by the people experiencing them.

METR's randomized trial (arXiv 2507.09089) is the cleanest of the four. Experienced open-source developers working on their own repositories ran 19% slower with AI assistance while believing they were 20% faster, after expecting a 24% speed-up going in. That is a ~39-point gap between measured time and self-report, under randomization. The randomization is what moves it past anecdote.

The Distribution Matters More Than the Average

Brynjolfsson's field experiment reports a 14% average productivity lift. That average is the least useful number in the study. Decompose it and you get +34% for novices, near-zero for veterans. The mean tells you to roll out uniformly. The distribution tells you it is an onboarding accelerant, not a senior-IC multiplier.

Dell'Acqua's BCG study adds the second axis, the jagged frontier. Inside it, consultants did 12.2% more tasks, 25% faster, at higher quality. One task outside it, with no visible boundary, ran 19% less likely to be correct. The lift is conditional on the task, not just the user.

A pooled average across a mixed population is a number that describes no one on your team.

The Enterprise Reality Check

MIT NANDA's field study (150 interviews, 350 surveys, 300 deployments) carries the sobering figure: 95% of enterprise GenAI pilots delivered zero measurable P&L impact. The blocker was workflow integration, the 'learning gap,' not model quality. Vendor-bought solutions succeeded at ~67% against ~22% for internal builds. ROI concentrated in back-office automation, not the sales and marketing tools that consumed most of the budget.

Glean's 6,000-worker survey supplies the mechanism. AI saves time, and much of it goes back into cleanup and rework. A naive completion-rate metric reads high because it does not instrument that downstream cost.

What This Changes

Surveys, NPS, and 'felt faster' metrics carry a known upward bias that METR quantifies at 39 points. The thing those metrics don't tell you is actual completion time. The replacement is a measured holdout, stratified by skill tier and task-frontier alignment, instrumenting completion time and quality. The holdout costs little next to redirecting a team on pooled averages that were measuring noise.

Two Teams Shipped What Most Are Hand-Waving

Wix and Dropbox both published concrete agent evaluation methodologies this cycle — not frameworks, not position papers, but measured ablations with reproducible designs. Combined with Meta's reward-model diagnostic and the OpenClaw red-team experiment, you now have a copy-paste playbook for the problem most teams solve with eyeball checks.

Wix: The Docs-First Decision Rule

Across 250 evaluations comparing agent-optimized docs vs. skills (cached tool outputs), docs lifted CLI task completion from 67% to 87% and cut token use 35%. On API tasks both tied at 80% completion, but the tradeoffs inverted: docs ran 31% faster; skills used 29% fewer tokens.

The decision rule is clean: docs as foundation, skills as an evaluated caching layer. Skills only win when validated and fresh. The unvalidated version — a cache that returns stale or wrong responses — is the failure mode everyone skips past. It doesn't show up in aggregate accuracy until you slice by the queries that hit it.

Steal Wix's three-metric harness: completion rate + tokens-per-task + latency over a fixed task set. Token efficiency isn't cosmetic anymore — it's a budget metric.

Dropbox: The Statistical Promotion Gate

Dropbox used DSPy with LLM-as-judge evals, human-labeled examples, offline replay, and statistical validation to convert agent evaluations into shipped Dash Chat improvements. The result: fewer incomplete answers, broader intent coverage, lower token use, no quality loss. What it replaces is shipping prompt changes on eyeball checks.

DSPy optimizes prompts as parameters rather than hand-tuning them. Worth running an ablation before you credit the full 35% token reduction to the framework — better retrieval and a tighter agent loop move the same number.

The Security Eval Gap: OpenClaw Shows the Way

Fernando Irarrázaval's public red-team (hackmyclaw.com) tested whether an assistant could be tricked into exfiltrating a secrets.env file. Over 2,000 participants sent 6,000+ prompt injections; the secret was never exfiltrated, at ~$500 in token cost. The methodology is the asset: plant a canary secret, build an adversarial corpus, measure exfiltration-success-rate as a release gate.

The Meta Diagnostic for Training Loops

If you run any RLHF/RLVR loop, Meta's paper proposes measuring both discriminative ability and specificity of the reward model, then using Monte Carlo dropout to cluster rewards into discrete, safer signals. This detects reward-model overreaction to equally-good responses — the precise mechanism that steers RL toward hacking. It's bolt-on instrumentation, not an architectural rewrite.