VectorRecallDrops40ptsat500KDocs;HybridHoldsUp

◆ DEEP DIVES

1. 01

   RAG at 500K Docs: The Recall Cliff Your Eval Doesn't See

   The Benchmark That Breaks the Demo

   Onyx released EnterpriseRAG-Bench this week: 500K+ synthetic documents spanning Slack, Gmail, Jira, GitHub, Confluence, Drive, HubSpot, Fireflies, and Linear, with misfiled files, near-duplicates, and conflicting versions baked in. The headline result is one every RAG architecture review this quarter should absorb. Dense vector retrieval drops from 90.7% recall at 5K documents to 50.6% at 500K. Same embedding model, same retriever, same queries. Corpus size is the only variable the experiment moves.

   A RAG benchmark that tops out at 10K docs is a sales demo, not an eval. The cliff arrives somewhere between 100K and 1M documents depending on topical density.

   Why BM25 Degrades More Gracefully

   The ablation worth staring at: BM25 falls only from 85.8% to 68.4% across the same range. A 17pp drop against dense retrieval's 40pp collapse. The mechanism is structural, not stylistic. As the corpus grows, embedding-space neighborhood density grows monotonically with it. Where 3-5 documents touched a topic at 5K, 40-60 touch it at 500K. The canonical answer gets crowded out of top-k by semantically similar but irrelevant neighbors.

   Retriever	5K docs	500K docs	Absolute drop	Failure mode
   Dense / vector	90.7%	50.6%	-40.1 pp	Neighborhood density
   BM25	85.8%	68.4%	-17.4 pp	Term overlap ambiguity

   BM25's failure mode (term ambiguity) is orthogonal to dense retrieval's failure mode (neighborhood density). That orthogonality is the cleanest argument for hybrid retrieval this quarter. Not because either retriever is best, but because their errors are uncorrelated.

   ---

   The Production Implication

   Two practical consequences fall out. First, fixed top-k is underfit for growing corpora. If top-k=10 worked at 5K docs where 3-5 documents were topically relevant, it fails at 500K where 40-60 are. Adaptive k, or aggressive cross-encoder reranking, becomes load-bearing. Second, any internal RAG demo benchmarked on fewer than 50K docs is systematically overstating production accuracy by 30-40 percentage points.

   Separately, Bing's engineering team published a framing distinction between search indexing (optimizing what a human should read) and grounding indexing (optimizing what an LLM should cite). Standard recall@k measures the former. Production RAG failures come from the latter. Retrieved passages that are topically relevant but evidentially insufficient for the generated claim. The eval harness needs a claim-level evidence sufficiency score sitting alongside retrieval metrics.

   What the benchmark doesn't tell you

   The corpus is synthetic. The embedding model and ANN configuration are unspecified. Whether late-interaction models (ColBERT) or Matryoshka embeddings compress the gap is open. The cleanest answer is an in-house ablation on the deployed corpus at deployed scale.

   Action items

   • Run your current retriever against EnterpriseRAG-Bench at 50K, 200K, and 500K scales — log recall@10/50 and MRR per scale tier
   • A/B hybrid retrieval (BM25 + dense + cross-encoder rerank) against dense-only on a 200K+ document slice
   • Instrument embedding neighborhood density (mean k-NN distance per ingest batch) as a production drift metric
   • Deprecate any internal RAG benchmark running on <50K docs as 'production-representative'

   Sources:Daily Dose of DS · AINews · MarketingShot

2. 02

   vLLM V1 Fixed Four Silent Bugs — Your RL Baselines Are Contaminated

   The Bugs That Don't Fail Loudly

   vLLM V1 shipped corrections for four issues that independently bias policy gradients in RL training. None of them crash the run. They are silent numerical contamination that shifts reward curves in a way you cannot distinguish from a hyperparameter sweep:

   V0 Issue	V1 Fix	What it breaks in RL
   Processed logprobs mismatched raw	Returns pre-processing logprobs	Off-policy correction ratios in PPO/GRPO are wrong
   Prefix caching on by default	Disabled in rollout paths	Cached prefixes produce stale logits → biased advantage estimates
   Weight-update model drift	Matched rollout and training weights exactly	On-policy algorithm silently becomes off-policy
   lm_head computed in bf16/fp16	Forced fp32	Logit tails shift; small but compounding in long rollouts

   Any eval or training run on V0 should be treated as suspect until re-baselined. This is the exact failure mode that makes RL papers irreproducible.

   Why This Matters More Than It Looks

   The logprob bug by itself is enough to invalidate PPO importance sampling ratios. The prefix-caching bug means advantage estimates were computed against stale state. Together, the reward curves teams liked from V0 runs may reflect infrastructure artifacts, not learning. A run that looked like it converged may have converged to the bug's attractor rather than the policy's.

   NVIDIA independently published lossless speculative decoding for RL rollout this week: 2.5× end-to-end speedup at 235B parameters without changing the policy distribution. The load-bearing word is 'lossless.' Speculative decoding normally biases sampling at temperature > 0, which is why it was previously unsafe for on-policy RL data. If the claim holds under replication, the combination of V1 fixes and lossless spec-decode makes RL post-training both correct and fast for the first time in a production stack.

   ---

   The Broader Harness Problem

   The V1 fixes land alongside evidence that harness engineering alone swings agent benchmark scores 10-20 points on identical base models (tau2-bench). Terminal-Bench 2.1 patched 28 of 89 tasks and absolute scores moved up to 12 points. Rankings held. Magnitudes did not. The thing this doesn't tell you from a top-line score: internal leaderboards that don't log (model_version, harness_version) as a tuple are producing numbers that cannot be compared week to week.

   The contamination runs both directions. V0's bugs inflated some results and suppressed others. The only way to know which of your checkpoints survived is to replay against V1. If the ranking doesn't move, the migration still pays for itself in audit trail. If it does move, you needed to know before shipping another model card.

   Action items

   • Re-run the two most important RL training runs from last quarter on vLLM V1 and diff reward curves against V0 baselines
   • Add harness_version as a mandatory field in every internal leaderboard entry — rerun any model comparison from the last 90 days that didn't hold harness constant
   • Evaluate lossless speculative decoding for your RL post-training rollout loop — run an A/A on reward curves before trusting the A/B
   • Pin vLLM version in all training and eval Dockerfiles with hash verification

   Sources:AINews · TLDR AI · Techpresso · Engineer's Codex

3. 03

   Anthropic's Per-Result Pricing Breaks Your Cost Model

   The Token Model Is Dead for Agentic Work

   Anthropic is the first frontier lab to move off token-based metering, charging per delivered result instead of per token. In the same week: Claude Code Auto Mode (targeting 90% autonomous task completion), Agent SDK going GA, and Managed Agents shipping Dreams (trajectory replay), Outcomes (success-criteria grading), and Multiagent orchestration. The old eval question was cost per million tokens at a fixed quality bar. The new question is cost per successful outcome. The two almost never correlate cleanly.

   Dimension	Claude Code Auto Mode	OpenAI Agents SDK	Your in-house router
   Pricing unit	Per result (new)	Per token	Per token (passthrough)
   Model selection	Vendor auto-picks per task	Developer-specified	Hand-tuned heuristics
   Autonomy target	~90% (unverified)	Not stated	Varies
   Reproducibility	Non-deterministic	Deterministic if pinned	Deterministic

   The Sonnet→Opus advisor pattern from the Code with Claude event claims frontier-quality benchmarks at 5× lower cost, routing cheap calls through Sonnet and escalating to Opus conditionally. No escalation-rate disclosure, no task breakdown, no public eval harness. The thing this doesn't tell you is how well-calibrated the escalation decision is on your distribution. The economics only work if escalation rate is low and the router is right when it escalates.

   FinOps dashboards keyed on $/1M tokens are already stale. They need a second axis: $/successful-task. That requires task-level success instrumentation most teams don't have.

   ---

   What This Changes in Practice

   When the vendor auto-selects models per call, A/B tests that assume a fixed-model baseline produce noisy results. The same prompt hits different weights across runs. Log the chosen model per call, pin models for experiments, and move to variance-aware significance testing.

   The routing implication is concrete. Short, deterministic calls where token pricing is already near floor will look worse on outcome pricing. Multi-step agentic work where completion rate drives cost will look better, possibly much better. The only way to know which side your workload falls on is to run the A/B on real traffic.

   Rate limits doubled, not just pricing

   Claude Code 5-hour rate limits doubled across Pro/Max/Team/Enterprise, peak-hour throttling is gone, and Opus API ceilings are up, sourced to the 220K-GPU Colossus 1 lease. Any agent workflow that was throttled needs rebenchmarking. Last quarter's finding that "the agent gives up on long tasks" may have been measuring the rate limiter, not the model. I flagged that risk in the prior column. The fix is cheaper than the analysis we did last time.

   EU AI Act extension buys 16 months

   The high-risk deadline moved from August 2026 to December 2027. Keep the compliance roadmap, slow the hiring ramp, reclaim the budget for evaluation work. Connecticut SB5 (codifying that automated decisions are not a defense to discrimination) has a shorter fuse. Whistleblower triggers at 10^26 FLOPs go live October 2026.

   Action items

   • Rebuild LLM cost attribution dashboards to track $/successful-task alongside $/1M-tokens — start with a binary task_success label piped into telemetry
   • Spike the Sonnet→Opus advisor pattern against your top-3 LLM workloads — measure quality delta vs. pure-Opus on your existing eval set
   • Rebenchmark Claude Code throughput under new 2× rate limits — compare wall-clock and success@k versus April results
   • Log cumulative training FLOPs as a first-class metric in your experiment tracker — CT SB5 makes this a regulatory trigger at 10^26

   Sources:AI Weekly · Future Perfect · TLDR AI · TLDR · Techpresso · ben's bites

4. 04

   Rotate Now: Braintrust Breach + Four Critical ML-Stack CVEs

   The Eval Platform Had Your Keys

   Braintrust, the LLM eval platform most ML teams use to score prompts, agents, and RAG chains, confirmed unauthorized access to an AWS account holding customer API keys. If anyone on the team ever pasted an OpenAI, Anthropic, Bedrock, or Vertex key into a Braintrust project, which is the standard integration path, treat those keys as compromised. The monetization pattern on leaked LLM keys is well-understood by now: attackers burn them on inference against the victim's billing account, usually through proxy resellers. The thing the platform alert doesn't tell you is that detection lag is measured in days of spend.

   Four Simultaneous Critical CVEs in the ML Stack

   Component	CVE / Severity	Attack	Where it hurts you
   Apache Iceberg	CVE-2026-42812 / 9.9	Write table metadata to attacker location	Silent feature poisoning in training datasets
   DocsGPT 0.15-0.16	CVE-2026-26015 / 9.8	MCP test bypass → RCE	RAG/agent stacks with MCP endpoints
   NVIDIA NVFlare	CVE-2026-24178 / 9.8	AuthZ bypass + privesc	Federated learning trust boundary collapse
   Ollama <0.17.1	CVE-2026-7482 / 9.1	Heap OOB read in GGUF loader	Dev workstations running local LLMs

   Iceberg is the one to take seriously for data teams. Write access to the catalog is write access to the training data's ground truth. A compromised metadata path lets an attacker redirect feature materialization to poisoned tables, and no downstream query engine will flag it.

   If requirements.txt isn't hash-pinned and the Iceberg catalog isn't tightly ACL'd, this week's CVEs already walked through the ML stack.

   ---

   The Worm Is Cross-Ecosystem Now

   The Mini Shai-Hulud worm ran for roughly 48 hours on April 29-30. It compromised PyTorch Lightning on PyPI, SAP packages on npm, and intercom-client and intercom-php on Packagist. It stood up 1,800 attacker-controlled GitHub repositories from stolen credentials. The mechanism is credential reuse across ecosystems: lift a GitHub token from an npm install, pivot to PyPI and Packagist, republish. Fully automated. Copycats follow within weeks, because they always do.

   Separately, internet-exposed inference infrastructure is being actively enumerated. AIMap fingerprints MCP servers, Ollama, vLLM, LiteLLM, LangServe, Gradio, and ComfyUI through Shodan and Nuclei. If any of those stood up on a VPC with a public subnet, even briefly for a demo, assume the instance is already in someone's catalog.

   Action items

   • Rotate every LLM provider API key ever stored in Braintrust — check provider billing dashboards for anomalous usage in the trailing 14 days
   • Patch Apache Iceberg and tighten catalog write ACLs — add manifest hash verification to downstream feature materialization DAGs
   • Upgrade Ollama to ≥0.17.1 across all dev workstations and enforce GGUF provenance checks from internal registry only
   • Enforce hash-pinned lockfiles (pip-tools, uv lock) on all training and serving images — unpinned deps should fail CI

   Sources:SANS AtRisk · TLDR InfoSec · The Hacker News · CyberScoop