Sharding for 100x growth

Database sharding is the practice of splitting one large database into smaller, separate databases so reads and writes can spread across more machines. (learn.microsoft.com) Microsoft’s Azure Architecture Center says a single database server eventually hits limits on storage, compute, and network bandwidth, even after teams add more memory, processors, or disks. Sharding moves past that ceiling by dividing data into horizontal partitions, with each shard holding a distinct subset of rows under the same schema. (learn.microsoft.com) The thread tied to this story frames sharding as a playbook for “100x growth,” a target that mirrors real production planning at Meta. In a December 19, 2023 engineering post, Meta said Threads had to prepare for “100x growth” and that the app reached 100 million sign-ups in its first five days after launch on July 5, 2023. (engineering.fb.com) The core design choice in any sharded system is the shard key, the field that decides where each row or document lives. MongoDB’s documentation says the shard key determines data distribution, and that a poor choice can create uneven chunks and overloaded nodes. (mongodb.com) A simple example is user-based sharding: every record for user 12345 goes to the same shard, while user 67890 lands on another one. Vitess, a MySQL sharding system, routes queries to the right shard by calculating the sharding key for each query, which lets applications start unsharded and later split data horizontally with little or no downtime. (vitess.io) That is why most sharding guides start with smaller steps before a full split. Vitess documentation describes vertical sharding as moving tables out of one unsharded keyspace into another, then horizontal sharding as splitting the rows inside a keyspace across multiple shards. (vitess.io) PostgreSQL draws a similar line between keeping one logical table and breaking it into smaller physical pieces. Its current documentation says native partitioning splits a large table inside one database, which can improve maintenance and query pruning, but it is not the same as distributing that table across multiple database instances. (postgresql.org) The tradeoff is complexity. Microsoft says sharding requires routing logic that sends each request to the right shard, while MongoDB’s manual adds that sharded clusters depend on shard keys, chunk movement, and query routing working together to keep data balanced. (learn.microsoft.com) (mongodb.com) The practical lesson in the thread is not “shard everything first.” It is to pick a shard key that matches real query patterns, isolate the busiest data paths, and leave room to reshard before one machine becomes the bottleneck you cannot scale past. (mongodb.com) (vitess.io)