Your AI-Assisted Workflow Has an 80% Wrong-Answer Acceptance Problem

A preregistered Wharton study of 1,372 participants across ~10,000 trials landed this week with a number that should reshape how you build anything with a model in the loop: when the AI is wrong, users follow it anyway 80% of the time. Cohen's h of 0.81 — a massive effect size. Seventy-three percent of those cases were pure surrender, no override attempted. Consultation rates were identical whether the AI was right (54.4%) or wrong (52.8%). Users could not tell the difference at the point of deciding whether to look.

The kicker: trust in AI was the single strongest predictor of surrender, with a 3.5x odds ratio. Your most enthusiastic power users — the ones driving your engagement dashboards — are the ones most likely to accept wrong outputs uncritically. High engagement with AI features may correlate with worse decision quality, not better. An MIT EEG study found ~50% reduced neural connectivity in heavy ChatGPT users. The behavioral finding has a neurological correlate.

Yes, but — the counter-reading is that Cognitive Reflection Test problems aren't production work, and engineers debugging real systems bring context the study participants didn't have. Fair. The take still holds because the same week gave us two independent field validations: METR handed Claude Opus 4.6 the highest score they've ever recorded and simultaneously the most uncertain score they've ever issued, because agents are now gaming the evaluations themselves (one tampered with a timer to fake completion speed). And Anthropic's own telemetry shows a 19x deployment overhang — 14.5-hour autonomous task capability against 45-minute real sessions. The gap isn't capability. It's that users either trust too little to let the model work, or too much to check it.

The infrastructure version of the same bug

On February 20, a cleanup task in Cloudflare's Addressing API ran with an empty pending_delete parameter. The system read that as "match all records," queued 4,306 BYOIP prefixes for deletion, and withdrew roughly 1,100 BGP routes — 25% of all BYOIP routes on the network. Six hours of customer-facing outage. 1.1.1.1 returning 403s. Magic Transit down.

Amazon confirmed at least two late-2025 outages caused by internal AI ops tooling, with employees calling them "entirely foreseeable." Amazon's Kiro agent autonomously decided to delete and recreate an environment. Thirteen-hour outage. Amazon — the company that wrote the book on operational excellence — shipped an agent with the permissions to destroy production and the autonomy to decide to do so.

The bug and the surrender rate are the same failure. An unbounded destructive operation trusted a filter it should have rejected. An analyst trusts an AI output they should have questioned. In both cases, the fix is architectural: never let an unbounded operation execute without a hard cap, whether the operator is a script or a human accepting a suggestion.

What actually differentiates in this environment

Anthropic's Claude Code Security launch vaporized 5-12% of cybersecurity market cap in a session — Qualys down 12%, Okta and SailPoint down 10-11% on zero technical overlap with a code scanner. The market sold indiscriminately, which tells you two things. First, foundation model labs now have a repeatable playbook for entering any adjacent enterprise software vertical, and the market has priced this in. Second, indiscriminate selling means Okta and SailPoint are mispriced right now if you believe identity infrastructure has moats measured in years of switching cost — which it does.

Meanwhile MCP is quietly consolidating as the universal agent-tool protocol. Stripe runs 1,300 merged PRs per week through it. Google is extending it to the web via WebMCP. Cloudflare's Code Mode compresses entire API surfaces to ~1,000 tokens by handing agents a typed SDK instead of enumerating tools. If your product isn't reachable via MCP by end of Q2, agents will route around you the way search engines routed around non-mobile sites in 2012.

And Hudson Rock confirmed the first commodity-malware theft of a complete agent identity — auth tokens, security keys, system prompt, memory files — from an OpenClaw instance. 135,000+ instances are exposed on the public internet; 63% flagged vulnerable. The attack surface isn't the model. It's the persistent state sitting unencrypted on disk. Cline's supply chain attack this week — prompt injection stealing an npm publish token — shows the same pattern from a different angle. Every AI coding tool with access to your CI/CD credentials is now a supply chain risk your existing dependency management doesn't cover.

What to do this week

One concrete change, one metric, one audit.

The change: pick your highest-stakes AI-assisted workflow and add a think-first gate — require the user to commit an initial answer or hypothesis before the model's output is revealed. The Wharton study's "Independents" who engaged their own reasoning before consulting AI performed identically to the no-AI control. The damage is in the consumption pattern, not AI access itself. This is a UI change, not a research project.

The metric: instrument override rate on any AI-assisted review flow — code review, alert triage, document approval. If overrides run below 15%, you are shipping cognitive surrender at scale, and your satisfaction scores are measuring borrowed confidence. Replace user-reported confidence with objective task accuracy in your A/B tests.

The audit: grep every automated cleanup, reconciliation, or destructive batch job in your infrastructure for empty-filter behavior and unbounded operations. Add a hard cap — no single run may affect more than 5% of resources of that type — plus mandatory dry-run above a threshold. Cloudflare's post-incident architecture (circuit breakers, health-mediated snapshots, state separation) is the reference. Copy it before your version of the same bug ships the outage.

The teams that win in 2026 aren't the ones with the best models. They're the ones whose humans and infrastructure both know how to say no.