Qwen3-4B Climbs 32% to 72% on Random Noise, No Training

The mechanism, and why the direction of the noise is irrelevant

The proposed pathology is autoregressive lock-in: the first ~20 tokens set a formatting template, and the model spends its remaining budget servicing it. The token counts back this up. Correct arithmetic runs averaged 718 tokens; wrong runs hit the 1,024-token ceiling and truncated mid-strategy. Two random vectors at positions 0-1 perturb the opening trajectory enough to break the template.

The ablations, reported via Artificial Intelligence Made Simple's coverage of the Irys work, close off the boring explanations:

Optimized projections did no better than pure noise (Mann-Whitney p=1.000), and Euclidean versus hyperbolic mutations were indistinguishable. That points to an energy-based effect — stochastic resonance, where added randomness helps a system escape a stuck state — not a prompt carrying information. The paired significance test on the 2-token arithmetic result was McNemar p=0.000015.

The envelope is narrow, and two gates will bite you

4-bit quantization nearly eliminates the effect. Qwen3-8B gained +12.8pp at 8-bit but a null +1.3pp at 4-bit (p=0.000177); the working hypothesis is 256 versus 16 distinct weight values, too coarse a grid for the perturbation to move anything. Models near ceiling regress: DeepSeek-R1-Distill-1.5B went 76.0% to 74.4%. Dose-response peaks sharply at two prefix tokens and decays from three to eight as the model branches into half-finished strategies. The role also changes by size: on 4B it is a convergence aid (answer-anywhere barely moves, 80% to 82%, but stated accuracy jumps), on 8B an exploration aid (answer-anywhere 32% to 50%).

The takeaway that needs no noise at all

Across 500 generations on both 4B and 8B, every naturally-terminated response was correct: P(correct | natural EOS) = 1.000. If that replicates on your task distribution, it is a confidence gate you can ship this week. Route truncated generations to resample or fallback instead of returning them. It costs one field in the inference logs.

Then the selector. Oracle best-of-10 reaches 100% on 4B, plurality voting realizes 72%, and the mean single seed is 51.6%. Majority voting is actively harmful when per-seed accuracy sits below 50%: 40% on 4B and 12% on 8B, the latter below the 16% baseline. Switching majority to plurality and gating on EOS are two config changes that recover more than any plausible base-model upgrade in the same period.

The modeled economics: ten seeds at roughly $0.009 per query against ~$0.45 for a frontier thinking model, with the two prefix vectors adding 0.097% overhead. At 10K queries/day that is $2,700 versus $135,000 a month. Discount the vendor-flavored arithmetic heavily; the ratio still justifies auditing which workloads overpay.

What the caveats forbid

The things these numbers don't tell you are load-bearing. Samples are tiny (25/5/12 tasks), n=3 scout runs overestimated the effect by ~9pp, the legal scorer was broken on 9 of 12 tasks, and a prior verbosity claim turned out to be a code bug. The failure that matters: on out-of-knowledge tasks such as GDPR classification and data-breach triage, every condition fabricated content.

Noise adds energy, never knowledge — it helps a model finish a computation it already did, and does nothing for one it never could.

The proxy is leaking, and the field is conceding it

The most telling response to the coding-agent bug reports is not a defense. It is two new benchmarks. SWE-Marathon and Frontier Code extend evaluation across multiple PRs and longer horizons, which is the field admitting that SWE-bench-style single-shot correctness does not measure the bottleneck teams actually hit. The Institute for Ethical AI & ML's read on HumanLayer's field reports is that models are trained and scored on short-term functional correctness, so maintainability regressions arrive invisibly. That report is qualitative — no sample size, no control group, no significance test — so treat it as direction, not effect size. The honest instrumentation is cheap: bugs- and incidents-per-developer, measured before and after you widen agent merge permissions.

The same failure, four domains

The Bear Cave's Axon item is the one to keep in a design review. It is a deployed generation system in a legal-evidentiary domain, marketed on time savings, and public records show it getting facts wrong. ROUGE and user-acceptance rate are silent on hallucination. They measure fluency and convenience. The gate that survives cross-examination is claim-level groundedness, NLI-style entailment of each generated assertion against the source document, reported alongside every release rather than once at launch.

Morning Brew's AppZen figures are the statistics lesson. The 3.5M+ AI-generated receipts produced on a handful of sites in six months is a lower bound on generation capacity, and it is real. The ~30% rise in detected fraud from 2024 to 2025 is a base-rate trap. Detection count equals model sensitivity times submission volume times true prevalence, so a better detector alone prints that number with zero change in underlying fraud. Only a fixed-denominator, randomly-audited holdout separates "our model improved" from "the threat grew."

Where the sources actually agree

Four independent write-ups converge on the same diagnosis. The scoring layer, not the model, is where value leaks. They differ sharply in rigor, and that matters for how you cite them. Meta's GAMUT, described by TheSequence, is the only one offering a method rather than an anecdote: a two-level meta-rubric that converts structured requirements into binary checklist criteria for LLM judges instead of holistic 1-5 scores. Binary decomposition is the standard variance-reduction move in human annotation, and it should transfer to judge models. The reporting includes no agreement statistics, so treat it as a design pattern to A/B against your current judge, not a validated improvement.

The distribution-shift point underneath the receipt story generalizes to any document model you run. Legacy detectors leaned on artifacts of physical capture: scanner noise, phone EXIF, JPEG recompression, template drift. Diffusion-generated documents are clean and internally consistent, so a content-rules pipeline degrades silently, with no alert firing. Forensic, provenance, and behavioral-graph features are the harder-to-spoof replacements.

A detector tuned before synthetic inputs scaled will keep reporting its old accuracy, because it is still being tested on the distribution it was built for.