Debate: Kafka overkill for low traffic
A social thread pushed back on using Kafka for low‑traffic event streams—examples cited systems handling roughly 40 orders per minute—and argued simpler architectures can be more stable and easier to operate. The discussion highlights tradeoffs between operational overhead and scale‑ready platforms. (x.com)
A debate over Apache Kafka flared after a social post argued that systems handling about 40 orders a minute do not need a heavyweight event backbone. (x.com) Apache Kafka is software for moving event streams — records of things that happened — between producers and consumers, while also storing those records and replaying them later. The Apache project says Kafka is built for high-performance data pipelines, streaming analytics, data integration, and other mission-critical applications. (kafka.apache.org 1) (kafka.apache.org 2) The post that set off the discussion used a low-volume example, roughly 40 orders per minute, to argue that a database table, job queue, or direct service call can be easier to run than a distributed log. The thread spread as engineers compared the staffing and operational cost of running brokers, topics, partitions, and consumers against the traffic actually on hand. (x.com) Kafka’s design assumes operators may want multiple copies of data across servers so the system keeps running when a machine fails. Apache Kafka documentation says a common production setting is a replication factor of 3, and Confluent’s design docs describe topic data being replicated across a configurable number of servers. (kafka.apache.org) (docs.confluent.io) That architecture can be useful when one stream feeds many downstream systems, when consumers need to replay history, or when event rates can climb far beyond today’s load. Apache and Confluent both describe Kafka as a platform for durable publish-and-subscribe messaging, long-term event storage, and stream processing at scale. (kafka.apache.org) (docs.confluent.io) Engineers arguing for simpler setups pointed to tools that do less. RabbitMQ’s documentation centers on queues, exchanges, and reliable message delivery, while Amazon Simple Queue Service is marketed by Amazon Web Services as a fully managed queue for decoupling and scaling distributed applications. (rabbitmq.com) (docs.aws.amazon.com) Those alternatives also change who carries the operational burden. Amazon Web Services says Simple Queue Service is fully managed, and RabbitMQ tutorials present a narrower message-broker model than Kafka’s combination of log storage, replication, and stream processing. (docs.aws.amazon.com) (rabbitmq.com) (kafka.apache.org) The pushback to the post was not that Kafka is unnecessary everywhere, but that teams often adopt scale-first infrastructure before they have scale. Kafka’s own documentation and Confluent’s tuning guides focus on throughput, partitions, batching, replication, and benchmarking — tools that matter most when volume, fan-out, or replay requirements are already real. (kafka.apache.org) (docs.confluent.io) The thread landed because it turned a familiar architecture question into a concrete one: if the system only sees dozens of orders a minute, the harder part may be operating the platform, not moving the messages. (x.com)
