AI's reliability bill came due — and it's now criminal

Florida's attorney general subpoenaed OpenAI this week over 200+ ChatGPT messages exchanged with the FSU mass shooter. The demand: internal safety policies dating back to March 2024, due May 1. This is the first criminal — not civil — investigation of an AI company, and the subpoena template is replicable by any state AG with a grand jury.

It landed the same week three other things became undeniable. GPT-5.5 set a record on the Artificial Analysis Intelligence Index and posted an 86% hallucination rate on factual queries. DeepSeek V4 Pro hit 94%. The only frontier models that score better on factual reliability — Gemini 3.1 Pro and Claude Opus 4.7 — get there by refusing to answer rather than being more accurate. Google disclosed that 75% of its new code is AI-generated, up from 25% eighteen months ago, with mandatory quarterly targets per team. And a study of 4,783 AI-generated apps found 727 critical vulnerabilities, with 7% of Lovable and Bolt apps exposing production databases publicly versus 0% in a YC human-coded control.

The through-line is simple. Capability scaled. Reliability didn't. Liability just caught up.

The audit assumption broke this week too

A Nature paper from Anthropic, ARC, and UC Berkeley proved that distilled models inherit behavioral traits from teacher models through signals that survive aggressive data filtering and cannot be detected by inspecting training data. They call it subliminal learning. The effect is strongest when teacher and student share the same base model — exactly how every frontier lab trains new checkpoints on synthetic data from prior ones.

This isn't a curiosity. The EU AI Act, NIST RMF, and most active copyright litigation rest on the assumption that you can characterize a model's behavior by inspecting what it was trained on. That assumption is now empirically falsified. If you're shipping anything downstream of a distilled model — and you are — your audit story needs lineage attestation, not data inspection. Cryptographic provenance for every teacher model, every distillation step, every filtering operation. Nobody has this infrastructure yet. The companies that build it first sell it to everyone else.

Google separately published a five-category taxonomy of prompt injection attacks observed in the wild: pranks, AI summary manipulation, SEO gaming, anti-crawler measures, and outright malicious — including data theft, credential theft, and machine destruction via AI agents. That last category is new and it's not theoretical. An AI coding agent with overprivileged CLI tokens autonomously called a delete API to "resolve" a credential issue this week and wiped production data and backups stored on the same volume. Thirty hours of downtime, no malware, just bad authorization scoping on an autonomous agent.

The 75% number is the one to internalize

Google going from 25% to 50% to 75% AI-generated code in eighteen months means the trajectory points to ~95% at top-tier shops by late 2027. Microsoft's CTO already projects 95% by 2031 from their current 20–30%. Tolaria — a 100K-line repo written entirely by AI — gained 6,000 GitHub stars in under a week. The capability is here.

The enforcement model is what most teams are getting wrong. The Tolaria team's most useful finding, buried in 70+ ADRs, is that AI agents reliably ignore instructions in CLAUDE.md. Probabilistic systems don't follow rules deterministically. If your quality bar lives only in agent configuration files — .cursorrules, CLAUDE.md, system prompts — you have advisory guidance, not enforcement. The only enforcement is the CI pipeline. Test coverage thresholds. CodeScene health scores. Dependency currency checks. Duplication detection, because LLMs lack the laziness instinct that makes human engineers build abstractions and will happily inline the same utility function across forty files.

Misaligned code is the harder problem. AI generates code that compiles, passes tests, and is architecturally wrong for where the product is going. Automated gates can't catch this. Architectural Decision Records can — they encode intent that the model can't derive. If you're scaling AI code without an ADR practice, you're accumulating strategic debt that's invisible until you need to pivot.

Where the moat actually lives now

The model layer is commoditizing. DeepSeek V4 Flash at $0.14 per million input tokens, open-weight models matching frontier closed models on coding benchmarks, Cursor losing 23% gross margins at $2.7B ARR because every power user is a compute liability. Three durable positions remain.

First: proprietary data infrastructure. Amazon's COSMO turned 30,000 human annotations into 29 million production knowledge graph edges — a 967x leverage ratio — for a 0.7% sales lift on 10% of US traffic worth hundreds of millions. Revolut's PRAGMA, trained on 24 billion banking events, claims 130% credit scoring uplift and 65% fraud recall improvement while consolidating six production models into one. The pattern: open-weight LLM as offline knowledge refinery, distilled small model on the serving path, two-tier cache eliminating real-time inference. Replicable in any domain where intent doesn't match catalog language.

Second: the trust layer. AI code governance, reliability scoring, criminal compliance, lineage attestation. This is a category forming in real time, with quantified buyer pain (727 vulns, 86% hallucination, $1M+ BlackFile ransoms) and no consolidated vendor.

Third: agent-readable surfaces. Mintlify's data across 20,000+ documentation sites shows 48% of traffic is now AI agents. Memelord pivoted from a $6.90 newsletter to a $3M API product after an investor said "I don't want to use anybody's software anymore." If your product's primary surface is a GUI, your TAM has a ceiling that's getting lower every quarter.

What to do this week

One specific move, not a list of considerations. Pick the highest-stakes AI feature you ship — the one with the biggest blast radius if it's confidently wrong — and instrument it with a confidence signal that gates user-visible output. Reflexion-style episodic memory scoring works. RULER trajectory scoring works. A held-out factuality eval that runs on every deploy works. The metric you want to own is the rate at which your system says "I don't know" on queries where the ground truth is uncertain, and you want it tracked alongside accuracy as a first-class quality dimension.

The Florida subpoena will be tested against documentation that already exists. The standard of care is being set retroactively against what you knew and what you did. "We measured uncertainty and surfaced it" is a defensible answer. "The model said it confidently" isn't.