Distributed systems primer
Recent social threads distilled core distributed‑systems patterns—load balancing, sharding, leader‑follower replication, service discovery, CAP tradeoffs—and argued engineers must design with real production load in mind rather than just CRUD cases. The conversation highlighted the need to separate builders focused on basic features from teams reasoning about latency, QPS, and fault domains. (x.com 1) (x.com 2)
A distributed system is one app spread across many machines, and the hard part is keeping those machines fast, in sync, and alive when one fails. (mit.edu) The basic scaling tool is load balancing: a front door that spreads incoming requests across many servers instead of sending every user to one box. Google Cloud says its load balancers can present a single Internet Protocol address while routing traffic to multiple backends. (cloud.google.com) When one database can no longer hold the traffic, teams shard it, which means splitting data into smaller partitions and storing each partition on different machines. That raises throughput, but it also means the application has to know where a user, order, or message lives. (abdullahslab.com) When one copy of data is not enough, teams replicate it by keeping multiple copies on different nodes so one failure does not take the service down. Google’s Site Reliability Engineering book says distributed consensus is used to agree on shared state such as leaders, group membership, and committed queue entries. (sre.google) A common replication pattern is leader-follower: one node accepts writes, then followers copy the leader’s log and serve some reads. That simplifies conflict handling, but it can add lag between the latest write and what a follower returns. (sre.google) Once services are split apart, they also need service discovery, which is the directory that tells one service where another service is running right now. Without that directory, autoscaling and failover break because addresses change as instances start and stop. (docs.cloud.google.com) The tradeoff that frames all of this is the CAP theorem, first proposed by Eric Brewer in 2000 and formalized by Seth Gilbert and Nancy Lynch in 2002. Their result says that during a network partition, a distributed system cannot guarantee both perfect consistency and full availability at the same time. (mit.edu) (acm.org) That is why production design starts with numbers, not diagrams: request rate, latency budgets, failure domains, and recovery targets. Amazon Web Services’ resilience guidance says architects have to choose tradeoffs around consistency and availability before failures happen, not during the outage. (docs.aws.amazon.com) The practical divide is between software that only needs basic create, read, update, and delete operations and software that has to survive millions of requests, cross-region traffic, and broken links between machines. The patterns are old, but the lesson is current: systems fail at real load, not in the architecture diagram. (cloud.google.com) (mit.edu)
