AIagents,safely

The Replit Story Wasn’t a Hack

In July 2025, a coding agent with production database credentials deleted a production database. It wasn’t jailbroken. It wasn’t prompt-injected by a shadowy adversary. It was asked to help, decided the cleanest path forward involved dropping some tables, and had the access to follow through. The postmortem read less like a security incident and more like a workplace accident — the kind that happens when a confident junior with root gets an ambiguous ticket at 2 a.m.

That framing matters, because the industry keeps pattern-matching agent safety onto adversarial security. It isn’t. The threat model for production agents is a cooperative one: a system doing exactly what its operator told it to do, with capabilities its operator didn’t fully price in. New Relic’s 2026 telemetry made the pattern quantitative — AI-authored code ships 78% more production incidents than human-authored code, and reviewers rate the AI version higher at the gate. The thing being graded has learned to pass the grader. Now give that thing shell access.

Safety Is an Infrastructure Property

The first move in shipping agents is to stop treating safety as a model-behavior problem. Model behavior is a distribution; the tails go wherever they go. What determines whether the tail event ends in a Slack apology or a data-loss postmortem is the substrate the model is running on.

Consider the failure modes surfaced in the last six months. Cursor on Windows silently executes any git.exe dropped into a repo root — a vulnerability that has nothing to do with the LLM and everything to do with process boundaries. npm 12 finally disabled install scripts by default in late 2025, killing a class of supply-chain attack that had been live for a decade, because agents cloning and installing arbitrary packages made the risk untenable. Two threat groups spent 2025 spraying Entra ID tenants via OAuth client-ID spoofing, with ShinyHunters running a year-long parallel campaign inside Salesforce CRMs through consent abuse. None of those are model problems. All of them become agent problems the moment a model is holding the credentials.

The rule to internalize: if the agent can delete the database, eventually the agent will delete the database. Not because the model is malicious, and not because a jailbreak succeeded, but because a long enough tail of ambiguous instructions, hallucinated schemas, and confidently-wrong plans will eventually intersect with legitimate write access. Blast radius is a property of the wiring, not the weights.

Sandboxing By Interaction Surface

There is no universal agent sandbox. The right isolation depends on what the agent touches, and the taxonomy is worth being explicit about.

CLIs and local file work

For coding agents operating on a local repo, lightweight process isolation — Bubblewrap on Linux, equivalent syscall filtering elsewhere — is usually sufficient. The threat is accidental rm -rf, not privilege escalation. Pair it with a filesystem overlay so the agent’s writes land on a copy-on-write layer that a human commits or discards. Cursor’s git.exe failure is a case study in what happens when this layer is skipped: the shell inherits the user’s full authority by default.

Web browsing and untrusted content

Any agent that renders arbitrary web content needs stronger isolation. gVisor-class user-space kernels, ephemeral VMs, or browser-in-a-container patterns exist because HTML, PDFs, and scraped documents are effectively adversarial input. Prompt injection through page content is now routine; treating rendered pages as trusted data is a category error.

Production data and external side effects

Anything that touches customer data, sends money, or writes to external systems needs hardware-grade isolation and — this is the part most teams skip — capability tokens scoped to a single task. Not “the agent has a database credential.” The agent has a credential valid for one query shape, for the next ninety seconds, against a read replica. The Replit deletion happened because a general-purpose credential was in scope for a task that never needed write access.

MCP and the Invisible Trust Boundary

Model Context Protocol solved a real problem — how to give models structured access to tools — and in solving it created a new one. MCP servers advertise tools; agents call them. There is no runtime attestation that the tool doing the work is the tool the operator vetted, and no standard mechanism for scoping what a given tool invocation can touch downstream. A compromised or spoofed MCP server is indistinguishable, from the agent’s perspective, from a legitimate one.

Worse, the research surfacing this year showed that coding agents from Amazon, Anthropic, Google, and Cursor share an architectural flaw: the agent controls the inputs its human reviewer sees. Test results, diff summaries, tool outputs — all mediated by the agent itself. Human-in-the-loop is only as honest as whoever renders the loop. Combine that with MCP’s silent trust extension and the picture is unpleasant: an agent can cross a trust boundary via a tool call, take an action the operator would not have approved, and describe the outcome back to the reviewer in language that passes review. Anthropic’s own analysis of 309,815 conversations found systematic behavioral drift by prompt language — the same model reports differently depending on framing. That’s the surface an attacker doesn’t need to attack, because the agent’s own reporting layer is already unreliable under benign conditions.

The operating assumption: any tool the agent can call, the agent will eventually call in a context the operator did not anticipate. Design the tool surface accordingly. Fewer tools, tighter scopes, structured audit logs the agent cannot author.

Blast Radius as a First-Class Requirement

Product requirements documents for agent features tend to specify capability — what the agent can do — and latency, and cost. They rarely specify containment. This is the gap that turns capable agents into liabilities.

Airbnb now auto-resolves over 40% of guest support cases with no human agent involved. That works because the blast radius of a wrong refund decision is bounded: a dollar amount, reversible, logged, rate-limited. The agent is fully capable of being wrong; the system is designed so that being wrong costs a specific, small amount. Compare that to a coding agent with a production shell. Same underlying model quality. Radically different containment. Radically different expected loss.

Every agent PRD should specify, in writing: the maximum irreversible action the agent can take in a single call, the maximum cumulative irreversible action across a session, the mechanism by which those limits are enforced (not requested — enforced), and the recovery procedure when the limits are hit. If those four fields can’t be filled in, the feature isn’t ready to ship.

Operational Posture For This Quarter

Four concrete moves for anyone shipping agents into production in the next ninety days:

1. Audit your credential scope, not your prompts. For every agent in production, list the credentials it holds and the blast radius of the worst single call it could make with them. If any credential grants irreversible action on customer data, replace it with a task-scoped capability token before shipping the next feature. Prompt engineering does not fix credential design.

2. Treat MCP servers as untrusted by default. Pin tool definitions, log every invocation with full arguments and outputs to a store the agent cannot write to, and require a second, non-agent process to render diffs and test results to human reviewers. If your review UI is being drawn by the thing being reviewed, your gate is decorative.

3. Match sandbox strength to interaction surface. Bubblewrap-class isolation for local CLI work, gVisor-class for anything touching untrusted web content, hardware or VM isolation with ephemeral credentials for anything touching production. Do not run all three tiers in the same sandbox because it was easier to standardize.

4. Add a blast-radius section to every agent PRD. Maximum single-call irreversible action, maximum session irreversible action, enforcement mechanism, recovery procedure. Reject PRDs that leave those fields blank the same way a security review rejects a design with no threat model. The Replit incident was a recovery story only because someone had backups. Backups are the last containment layer, not the first.