Anthropic Eval Lets Model Publish PyPI Malware to 15 Hosts

The loop that pays the attacker twice

The botnet clicks ads on sites the same operators control, which means those clicks can be made to convert. Those rows arrive in training tables as positives. A CTR or bidding model fit on them learns to bid harder into precisely the inventory the botnet occupies. Fraud spend becomes a training signal for more fraud spend, and the loop closes without anyone approving it. Most teams filter flagged invalid traffic out of billing and reporting, then never propagate that filter to the feature and label tables the training job actually reads. Checking which table the job selects from is a one-hour audit with a potentially large spend delta.

Which signals survive the spoof

The cost column is the one that survives the next adversary iteration. Sensor telemetry and duty-cycle features hold up not because forgery is hard in principle, but because forging them reduces the attacker's monetizable output. A TV box throttled to a human diurnal rhythm earns less. Declared device strings carry no such penalty, which is why they degrade permanently rather than cyclically. Demote them, and never use them as a standalone gate.

Monitoring has to become multivariate

A per-feature PSI or KS test with a 0.2 alert threshold cannot catch a mimicry population whose marginal distribution matches genuine market share. It returns near-zero divergence by construction, which is a property of the test, not evidence about the traffic. Joint constraints do catch it, and they need no labels. A session claiming a flagship handset should not simultaneously present landscape-locked 1920×1080 geometry, zero motion, proximity or battery telemetry, a 24/7 duty cycle, and ethernet-grade RTT stability. Those combinations are physically inconsistent and encode cheaply as rules. Ship them as a monitor, not as a model.

Attribution and the browser are downstream of the same problem

Last-touch and multi-touch attribution pass fraudulent conversions straight through, so a CAC computed from attributed conversions is not a measured quantity. Randomized geo-holdouts or ghost-ads designs are the only attribution methods structurally robust to click fraud, because a botnet clicking in both arms cancels in the difference. Separately, the Adform client-side script poisoning detected July 27, 2026 rewrote DOM content in visitors' browsers across downstream customer sites, with pre-detection dwell time undisclosed. Any event property computed in the browser through a third-party tag — order value, transaction ID, consent flag — is an untrusted model input until it is validated server-side.

The capability question nobody ablated

Unit 42 documented a threat actor running the open-weight DeepSeek model through the open-source Hermes Agent framework, triggered by a single Telegram instruction, to autonomously discover internet-facing targets and proceed with attacks. Read as a measurement, that is a data point about the capability floor, not the frontier. The report does not disentangle model contribution from scaffolding contribution, and gives no success rate, target count, or task-completion statistics. That distinction decides whether an agent roadmap chases better models or better scaffolds, and on this evidence nobody knows. The narrower claim holds and is still worth something: curated-benchmark task success does not measure loop robustness on open-ended objectives, and loop robustness is the property that bites in production.

A monitor that watches one feature at a time cannot see an adversary who copies the market's own distribution.

The anchoring failure moves two metrics in opposite directions

This is the cheapest experiment on the quarter's list, and its result determines whether any other number in the report can be trusted. When annotators see LLM-suggested labels, inter-annotator agreement rises, which reads on a dashboard as data quality improving. At the same time annotator errors become correlated with model errors. The gold set drifts toward agreement with the model it exists to evaluate. Track Cohen's kappa alone and neither effect appears.

Three arms on a shared item pool: blind labeling, suggestion-shown, suggestion-shown-with-confidence, expert adjudication as ground truth. Report kappa, error rate against adjudication, and the correlation between annotator error and model error. If error correlation climbs materially in the suggested arms, the offline metrics are optimistically biased and one real source of the offline/online gap has a name. A week of work, a defensible finding.

Most pilot verdicts are underpowered, not negative

The circulated claim that 95% of AI pilots show no P&L impact is unattributed and, as stated, nearly empty: no estimand, no observation window, no control arm, no minimum detectable effect. Typical enterprise pilots are single-arm, six to twelve weeks, and underpowered by roughly an order of magnitude to detect the 1–3% movement in a noisy revenue or cost series they implicitly test. Absence of a measured effect is not evidence of absence when the design could never have detected the effect. Some large share of that figure is measurement-design failure sitting on an unknown share of genuine value failure, and nobody has separated the two. Keep it out of planning documents. Compute the MDE for every pilot in flight instead.

The substrate your agents actually read

Turing Post's field findings explain why retrieval succeeds while reasoning fails. Several hundred 'published data views' at one client are dynamically generated JSON blobs rather than typed tables. Queryable, not composable: no joins, no contracts, no reliable text-to-SQL. Business-logic validation does not run at ingestion, so a bad partner number reaches an executive dashboard before a human sees it. That defect lands hardest on a modeling team, because upstream data errors become indistinguishable from real distribution shift in monitoring, and the drift alert fires on someone else's broken file. The warehouse program was scored on whether the data arrived. By that measure it succeeded.

Run it as an ablation rather than an argument. Partition published assets into typed, contracted tables versus semi-structured views, then evaluate the retrieval or text-to-SQL agent's task success on each partition with identical prompts. A wide gap is a defensible effect size for catalog and contract investment. A narrow gap saves a quarter. Two days either way.

The ownership gap, and the number not to lean on

The same phase keeps coming up short across the evidence reviewed here. Execution collapsed from months to minutes while alignment, specification and verification widened, and verification — eval harness, regression gates, drift monitoring, judged suites — has no department attached. A vendor survey reporting that 53% of organizations cannot fully verify what their AI agents do points the same way, but it is directional only: no n, no sampling frame, and the vendor sells the remedy. Cite the mechanism instead. The mechanism is that agent tool calls are not logged as structured, immutable, replayable events, which is also why a regression cannot be attributed to a prompt-version bump versus a model-version bump.

Built as a compliance artifact, an agent trace log comes out unusable for debugging; built as a debugging artifact, compliance falls out for free.