Amazon Kiro Agent Wipes Prod via Over-Scoped IAM Credentials

Do the arithmetic before the architecture. At 95% per-step accuracy, a 20-step chain finishes correctly about one time in three. Documented hallucination baselines run 3–27%, which makes the math worse, not better. A chain with no structural accounting for compounding error is a probability experiment. Production traffic is the sample.

Two incidents put a price on it. Amazon's Kiro agent was fixing a minor bug while holding operator-level credentials. It deleted a live Cost Explorer environment. The outage ran 13 hours. Separately, entity-binding-drift research shows what happens when an agent locks a wrong resolution early: wrong user ID, wrong file, wrong ticket. The error compounds up to 8.5x downstream. A cheap second-model recheck cuts that by 79%.

The sources agree on the diagnosis, and it is not model capability. Kiro did not need a smarter model. It needed a smaller IAM role. The countermeasure for compounding error is not a better prompt. It is deterministic control flow with a hard iteration cap, invoking the model only at the two or three points that need genuine judgment. The entity-drift fix is a verification checkpoint between steps, not a bigger model. Kent Beck's 2008 DARPA anecdote is the same mechanism: a car told to "relax constraints when stuck" drove onto a sidewalk. Agents execute what you encode, not what you meant.

The multi-agent debate, with Cognition's "Don't Build Multi-Agents" on one side and Anthropic's roughly-90%-better-at-15x-cost result on the other, resolved into one usable pattern. A single orchestrator owns full context and spawns isolated, short-lived sub-agents that each return a summary, with no peer-to-peer chatter. Treat the model as a stateless function. Conversation, plan, and progress belong in Postgres or Redis, not in the context window. That is what makes pause/resume and horizontal scaling possible. Stateless HTTP services taught the same lesson, and it holds here.

An agent's IQ does not matter if its IAM role does the damage. Scope the blast radius before you scale the model.

Standard evals sail past this mechanism, and the reason is simple. There is no token-level attribution in the output stream. Tokens generated from retrieved context and tokens generated from parametric weights arrive looking identical. Hallucination detection asks whether the model answered something entirely outside the corpus. That is not the common failure. The common failure is retrieval that was topically correct but incomplete, with the model quietly backfilling the gap from training data. That is partial-coverage leakage. It produces confidently wrong answers that pass a surface check.

A diagnostic session isolated it with a custom categorical rubric (corpus_abstention) using five fixed verdicts: GROUNDED, CORRECT_ABSTENTION, UNGROUNDED, MIXED_LEAKAGE, WRONG_ABSTENTION. Not a regenerating aggregate score. Fixed categories, so results compare across runs. The root cause turned out to be one instruction line. It let the agent skip retrieval for questions it "already knew." That line drove 6 of 15 in-corpus cases to smuggle in an unsupported claim. Remove the line, force retrieval on every query, and the eval moves from 19/33 to 30/33. Ungrounded answers hit zero.

The deeper structural point comes from a study of 12 agent memory systems. Reliability under fact updates and cross-session reasoning is decided at write time, not query time. Flat and append-only stores produce "hallucinations of the past": stale facts that survive updates because no schema exists to overwrite them. Typed graph memory does better. Graphiti's Pydantic ontology is the example, and it wins for an unglamorous reason: structure gets imposed while the memory is being written, not reconstructed later.

The caveat matters: graph memory in stateless, high-QPS agents costs real latency. It pays off when updates stay incremental. If every update rewrites the whole graph, it does not.

Partial-coverage leakage cannot be patched at query time if your retrieval or memory architecture never built structure in at write time.

Read this as a queuing problem. Arrival rate up, service rate flat. DX's Q2 report covers 500+ teams. AI-generated code jumped 34% to over 50% in a single quarter. PR throughput rose 37%. Median PR size nearly doubled. The composite Developer Experience Index fell 67 to 65. Run the queue math: 37% more PRs, each close to twice the size, feeding a review pipeline whose service rate did not move. Median PR size doubling is a known leading indicator of technical debt. It is not noise to shrug off.

Stripe's internal Minions orchestrator shows where the ceiling sits. One engineer fires 10-50 concurrent agents at bugs. The binding constraint is not whether the agent writes correct code. Stripe is past that. The constraint is review throughput, CI capacity, and merge-conflict resolution at 10-50x normal PR volume.

Two mitigations recur across the sources. First, cross-model review. Claude Code and Codex each miss the bug classes they themselves produce, and each reliably catches those classes in the other's output. That is ensemble logic from ML applied to CI. Reviewer diversity drives catch rate, not reviewer capability. Second, judge/test-gated migration. Anthropic's ~1M-LOC Zig-to-Rust port landed in under two weeks because a rigorous test suite existed to gate every diff. Remove the gate and agent output is fast and wrong. The gate is the mechanism, not the migration tooling.

The budget line sharpens the mandate. AI spend is up 28x against a flat innovation ratio. Adoption percentages will not survive a serious review. Before/after DXI-style data will.

AI got the code written much faster; if review wasn't redesigned to match, the bottleneck moved rather than disappearing.