xAI Open-Sources X's Grok-Based Ranking Stack on Apache-2.0

Why This Matters Now

xAI released X's complete For You feed recommendation system under Apache-2.0 — not a research demo, but the production system serving hundreds of millions of users. This is the most detailed open-source recommendation architecture since Twitter's 2023 release, and the key evolution is that a Grok-based transformer has replaced nearly all hand-crafted ranking rules with end-to-end ML.

Architecture Deep Dive

The system is organized into four components: Home Mixer (orchestration via gRPC), Thunder (in-memory post store with Kafka ingestion and sub-millisecond reads), Phoenix (ML retrieval + ranking), and a modular Candidate Pipeline framework. The codebase is 62.9% Rust, 37.1% Python — Rust handles serving and pipeline orchestration, Python handles model training.

The Multi-Objective Scoring Pattern

The ranking model predicts probabilities for 15+ distinct user actions — likes, replies, reposts, shares, follows, video watches, profile visits, plus negative signals like blocks, mutes, reports, and "not interested." The final score is a weighted linear combination: Score = Σ(weight_i × P(action_i)).

This decouples model training from product policy — you retrain to improve prediction accuracy, but tune weights to change feed character. Want less outrage? Increase the negative weight on "report" predictions. No retraining required.

The Attention Masking Trick

Each candidate post can only attend to the user's context — not to other candidates in the batch. This sacrifices cross-item modeling for two critical properties: deterministic scores (independent of batch composition) and cacheability per (user_context, candidate) pair. The diversity loss is compensated by a downstream Author Diversity Scorer. At X's scale, this is a massive latency and compute win.

Retrieval Architecture

Phoenix uses a two-tower model with dot-product similarity and multiple hash functions for embedding lookup. Thunder provides in-memory candidate sourcing with TTL-based retention and per-user partitions for posts, replies, reposts, and video.

What's Missing

There are no evaluation metrics, no A/B test results, no ablation studies, and no model size disclosures. We don't know the embedding dimensions, the actual weight values (arguably the most important parameters), or whether the Grok transformer outperforms a well-tuned gradient-boosted model. Treat this as an architecture reference, not a performance benchmark.

The Contamination Is Now Measured

OpenAI's internal audit found 60% of SWE-bench Verified coding tasks are either broken or compromised by model memorization. This is the benchmark the entire industry has used to compare coding model capabilities. OpenAI is calling for retirement in favor of SWE-bench Pro and private evaluation frameworks.

Simultaneously, multiple sources reveal a broader evaluation crisis:

• Anthropic's "Intelligence Yield" — measuring task difficulty per unit compute — is the right concept but has zero published methodology. No definition of task difficulty calibration, no compute measurement specification, no ablation studies.
• GLM-5 (744B MoE from Z.ai) is called "one of the most impressive open source models ever released" with literally zero benchmark numbers in the announcement.
• GPT-5 has a confirmed performance "dip" — but no specifics on benchmarks, task categories, or magnitude.
• GPT-5.3-Codex claims SOTA on SWE-Bench Pro with a 400K context window — but no pass@k, temperature, or comparison methodology.

The Credibility Hierarchy

Cross-referencing these signals with Confluence Labs' 97.9% on ARC-AGI-2 (a benchmark specifically designed to resist memorization), a clear evaluation credibility hierarchy emerges:

1. Internal held-out evals on your domain data — highest signal, hardest to contaminate
2. Contamination-resistant benchmarks (ARC-AGI-2, SWE-bench Pro) — designed to resist memorization
3. Public benchmarks with known contamination (SWE-bench Verified) — directional at best
4. Vendor-reported private benchmarks — lowest credibility without independent reproduction

The Deeper Problem

The shift toward private evaluation frameworks is the more concerning signal. If each lab evaluates on proprietary benchmarks, we lose independent model comparison entirely.

If 60% of your benchmark is memorized, you don't have a benchmark; you have a leaderboard for overfitting.

The METR developer productivity study collapse adds another dimension: their experiment was invalidated by selection bias because developers refused to participate without AI tools. If a well-funded research organization can't solve this measurement problem, your internal Copilot ROI analysis almost certainly can't either.