Kafka as a Task Queue Is Usually the Wrong Default

◆ DEEP DIVES

1. 01

   RabbitMQ vs Kafka vs Pulsar: The decision tree your team should actually use

   <h3>Three Different Machines, Not Three Options</h3><p>The persistent framing of RabbitMQ, Kafka, and Pulsar as competitors obscures the real insight: <strong>they solve fundamentally different problems</strong> with different data models. Choosing between them isn't a feature comparison — it's an architecture decision.</p><table><thead><tr><th>Dimension</th><th>RabbitMQ</th><th>Kafka</th><th>Pulsar</th></tr></thead><tbody><tr><td><strong>Core model</strong></td><td>Message broker (push)</td><td>Distributed log (pull)</td><td>Hybrid broker + log</td></tr><tr><td><strong>Message lifecycle</strong></td><td>Pushed → acked → deleted</td><td>Appended → retained by policy</td><td>Cursor-tracked in ledger</td></tr><tr><td><strong>Replay capability</strong></td><td>None after ack</td><td>Full via offset reset</td><td>Full via cursor reset</td></tr><tr><td><strong>Storage architecture</strong></td><td>Coupled to broker</td><td>Coupled (partitions on disk)</td><td>Separated (BookKeeper ledgers)</td></tr><tr><td><strong>Scaling model</strong></td><td>Add brokers</td><td>Add brokers + rebalance partitions</td><td>Scale compute and storage independently</td></tr><tr><td><strong>Operational complexity</strong></td><td>Low-medium</td><td>Medium-high</td><td>High (broker + BookKeeper + ZooKeeper)</td></tr></tbody></table><h4>The Decision Tree That Actually Matters</h4><ol><li><strong>Do consumers need to replay messages?</strong> No → RabbitMQ is likely sufficient and simpler. Yes → continue.</li><li><strong>Do multiple independent consumers need the same stream?</strong> Yes → Kafka or Pulsar. No → still consider Kafka for durability, but RabbitMQ may work.</li><li><strong>Do you need to scale storage independently of compute?</strong> Yes → Pulsar's architecture wins. No → Kafka's simpler operational model is worth the coupling.</li><li><strong>What's your team's operational capacity?</strong> Running BookKeeper + Pulsar brokers is meaningfully harder than Kafka with KRaft. Small platform teams should weight this heavily.</li></ol><h4>The Pulsar Nuance</h4><p>Pulsar's <strong>compute/storage separation via BookKeeper</strong> follows the same disaggregated pattern winning in modern databases — architecturally elegant and theoretically superior for independent scaling. But the operational tax is real: you're now running and monitoring BookKeeper clusters alongside brokers, <em>and Pulsar still requires ZooKeeper</em>. Meanwhile, <strong>Kafka's KRaft mode</strong> has eliminated its ZooKeeper dependency, meaningfully closing the operational simplicity gap.</p><blockquote>The most common anti-pattern: teams choosing Kafka because it's the default, then using it as a task queue with competing consumers and no replay — paying Kafka's operational complexity for RabbitMQ's use case.</blockquote>

   Action items

   • Audit your current messaging system's actual consumption patterns this sprint — specifically check whether consumers use replay, or if you're running Kafka as a glorified task queue
   • If evaluating Pulsar, run a proof-of-concept specifically testing independent storage scaling and mixed queue/streaming workloads against your Kafka baseline before committing
   • Document your messaging system decision rationale in an ADR (Architecture Decision Record) tied to your actual workload characteristics, not feature matrices

   Sources:EP203: RabbitMQ vs Kafka vs Pulsar

2. 02

   GraphQL's caching gap is worse than you think — and AI agents writing auth code is worse still

   <h3>The REST vs GraphQL Caching Reality</h3><p>The standard framing — REST gives server control with <strong>native HTTP caching</strong> (ETag, Cache-Control, CDN), GraphQL gives client control with a single flexible endpoint — understates the operational cost of that trade-off at scale.</p><ul><li><strong>GraphQL caching lives at the application layer.</strong> Persisted queries and response caching are bolt-ons, not primitives. You lose CDN offload for free, meaning your origin servers handle dramatically more traffic for equivalent read patterns.</li><li><strong>The complexity shift is asymmetric.</strong> GraphQL moves complexity from many clients to one server — which sounds like a win until that server becomes your gateway bottleneck. Resolver fan-out means your GraphQL gateway's <strong>p99 is bounded by the slowest downstream service</strong>.</li><li>REST's over-fetching problem is real but <strong>often cheaper to solve</strong> (sparse fieldsets, BFF pattern) than GraphQL's caching and complexity problems are to mitigate.</li></ul><p>The signal to move to GraphQL: when your backend team spends more time building one-off REST endpoints for different client needs than they would spend building and maintaining a schema. <em>That crossover point is higher than most teams think.</em> If you're building a single SPA talking to your own backend, REST with a BFF gives you the same flexibility with dramatically better caching.</p><hr><h3>AI Agents That Integrate Themselves</h3><p>WorkOS shipped an AI agent (<code>npx workos</code>) powered by Claude that <strong>autonomously reads your codebase, detects your framework, writes a complete auth integration, and self-corrects by feeding build errors back to itself</strong>. This signals a meaningful shift in developer tooling: from SDKs you integrate to agents that integrate themselves.</p><p>But authentication is the <strong>worst possible domain</strong> to trust to autonomous code generation. A subtle bug in token validation or session handling doesn't fail a build — it fails a pen test, or worse, a breach.</p><blockquote>Treat any AI-generated auth code as untrusted input requiring full security review. The pattern of self-integrating agents is significant; the domain they chose to demonstrate it in is concerning.</blockquote>

   Action items

   • If running GraphQL at scale, measure your actual cache-hit ratio gap versus equivalent REST endpoints this quarter before expanding GraphQL surface area
   • Establish a policy now: AI-generated authentication or authorization code requires mandatory security review before merge, regardless of the tool that produced it

   Sources:EP203: RabbitMQ vs Kafka vs Pulsar