NadMesh Harvests Cloud Keys from Exposed Ollama, ComfyUI

The study design is what makes the null worth taking seriously. Humlum & Vestergaard linked self-reported survey data to administrative earnings and hours records across roughly 25,000 workers, 7,000 workplaces, and 11 high-exposure occupations. Workers self-reported real savings — 2.8% of total work time, with 64–90% reporting some benefit — but the administrative data puts the estimate near zero, and the confidence interval rules out average effects above roughly 2%. That's a tight bound, not a shrug. The sample size is doing real work here, which is the part a smaller study couldn't have delivered.

Two explanations survive contact with this data, and they don't collapse into one another. There's a conversion gap, where the saved time got redirected into other tasks and never showed up as output. And there's a measurement gap, where the value flowed into channels — revenue, quality, avoided risk — that nobody instrumented. If you're building eval harnesses for production models, telling these two apart is not a side quest. It is the job.

The translation to eval practice is mechanical, which is the good news. For every production model, trace offline metric → workflow change → business KPI, and check that the terminal KPI actually shows up in telemetry. If it doesn't, you can't attribute value to anything, and you will reproduce the same false null the administrative records nearly produced. Pre-register a capture mechanism — throughput, cost, SLA, reallocation — before rollout. Efficiency without a designated sink diffuses. It doesn't concentrate on its own.

Two adjacent findings sharpen this. Multi-step agents compound error at a rate that single-turn accuracy hides entirely: a model at 95% per-step reliability fails roughly 40% of 10-step tasks. The benchmark that reports per-step accuracy is not measuring the bottleneck you'll hit in production — measure task-completion rate and trajectory correctness instead. Separately, the claim that engineering and research have fused is testable against this same logic: whoever owns the versioned, reproducible eval harness is the one whose hypotheses get tested against reality rather than against a leaderboard. Eval infrastructure is the throughput multiplier here, not overhead.

The honest caveat: a two-year post-ChatGPT window is plausibly underpowered against diffusion lag. Solow's paradox took roughly two decades to resolve, and nobody called that a failure of computers. Read this result as "measure the conversion," not "AI doesn't work" — the data supports the narrower claim, not the broader one.