Notion Shares Lessons on Scaling PostgreSQL
What happened
Notion's engineering team shared insights from scaling its core database to 480 shards without adopting a microservices architecture. The team credited its success to tightly controlling schema, sharding logic, and operational complexity, arguing that infrastructure simplicity can be more effective than distributed systems in early growth stages.
Why it matters
- The move to a sharded PostgreSQL architecture was a response to critical performance issues that emerged in mid-2020 as Notion's user base grew exponentially. These included frequent CPU spikes hitting over 90%, and the PostgreSQL VACUUM process failing to clean up disk space, which put the database at risk of a transaction ID wraparound failure that would force it into a read-only state. - Notion's core data model, where every piece of content is a "block," was central to the sharding decision. The engineering team chose to shard all tables transitively related to the main `block` table to avoid complex and inefficient cross-shard queries. - The sharding logic is handled at the application level within their Node.js monolith, rather than at the database layer. They use the `workspace_ID` as the partition key, which ensures that all data for a single workspace resides on the same shard, optimizing for the common user behavior of accessing data within one workspace at a time. - The migration to the sharded system was executed with zero downtime using a multi-step process. This involved double-writing new data to both the old and new databases, backfilling existing data, verifying data integrity, and then switching over to the new sharded database. - Initially, in 2021, the architecture consisted of 32 physical databases, each holding 15 logical shards, for a total of 480 logical shards. By 2023, to handle further growth, Notion increased its physical database instances to 96, with each containing 5 logical shards, maintaining the total of 480 logical shards. - This database sharding effort successfully reduced CPU and IOPS utilization from peaks of 90% down to around 20% during high traffic periods. - To gain deeper visibility into their database performance during and after the sharding process, Notion adopted the tool pganalyze in March 2021, based on a recommendation from PostgreSQL consultancy PGExperts. - The decision to shard the database while retaining a monolithic application architecture was a deliberate choice to maintain development velocity and data locality, which is crucial for their complex block-based data model.
Key numbers
- Notion's engineering team shared insights from scaling its core database to 480 shards without adopting a microservices architecture.
- - The move to a sharded PostgreSQL architecture was a response to critical performance issues that emerged in mid-2020 as Notion's user base grew exponentially.
- These included frequent CPU spikes hitting over 90%, and the PostgreSQL VACUUM process failing to clean up disk space, which put the database at risk of a transaction ID wraparound failure that would force it into a read-only state.
- Initially, in 2021, the architecture consisted of 32 physical databases, each holding 15 logical shards, for a total of 480 logical shards.
Quick answers
What happened in Notion Shares Lessons on Scaling PostgreSQL?
Notion's engineering team shared insights from scaling its core database to 480 shards without adopting a microservices architecture. The team credited its success to tightly controlling schema, sharding logic, and operational complexity, arguing that infrastructure simplicity can be more effective than distributed systems in early growth stages.
Why does Notion Shares Lessons on Scaling PostgreSQL matter?
The move to a sharded PostgreSQL architecture was a response to critical performance issues that emerged in mid-2020 as Notion's user base grew exponentially. These included frequent CPU spikes hitting over 90%, and the PostgreSQL VACUUM process failing to clean up disk space, which put the database at risk of a transaction ID wraparound failure that would force it into a read-only state. Notion's core data model, where every piece of content is a "block," was central to the sharding decision. The engineering team chose to shard all tables transitively related to the main block table to avoid complex and inefficient cross-shard queries. The sharding logic is handled at the application level within their Node.js monolith, rather than at the database layer. They use the workspace_ID as the partition key, which ensures that all data for a single workspace resides on the same shard, optimizing for the common user behavior of accessing data within one workspace at a time. The migration to the sharded system was executed with zero downtime using a multi-step process. This involved double-writing new data to both the old and new databases, backfilling existing data, verifying data integrity, and then switching over to the new sharded database. Initially, in 2021, the architecture consisted of 32 physical databases, each holding 15 logical shards, for a total of 480 logical shards. By 2023, to handle further growth, Notion increased its physical database instances to 96, with each containing 5 logical shards, maintaining the total of 480 logical shards. This database sharding effort successfully reduced CPU and IOPS utilization from peaks of 90% down to around 20% during high traffic periods. To gain deeper visibility into their database performance during and after the sharding process, Notion adopted the tool pganalyze in March 2021, based on a recommendation from PostgreSQL consultancy PGExperts. The decision to shard the database while retaining a monolithic application architecture was a deliberate choice to maintain development velocity and data locality, which is crucial for their complex block-based data model.
