Paxos' 21TB Cutover

Paxos published the engineering deep-dive on March 23, 2026, authored by Tom Wu, laying out the exact sequence their team used for the migration. (paxos.com) The team built a partitioned parent table around the live table, added a CHECK constraint proving the existing rows’ time range, attached the existing table as the “history” partition, and created new time-range child partitions for incoming writes. (paxos.com) To avoid an unpredictable blocking attach, engineers used ADD CONSTRAINT... NOT VALID and ran VALIDATE CONSTRAINT as a separate phase, which performs the unavoidable full-table scan under a SHARE UPDATE EXCLUSIVE lock and prevents autovacuum on the target table during validation. (paxos.com) That validation scan took just over nine hours in practice, causing P95/P99 tail-latency growth as dead tuples accumulated, and an initial attempt failed because write spikes exceeded timeout thresholds. (blockchain.news) On the retry Paxos coordinated with market makers to pause trading briefly and relaxed timeout thresholds to complete validation with P50 latency largely stable while higher percentiles degraded during the scan window. (blockchain.news) Engineers also resolved a partitioning gotcha: their system had relied on a BEFORE INSERT trigger to set a server-side timestamp, and once that timestamp became the partition key they had to ensure the key was available up-front for Postgres to route inserts correctly. (paxos.com) Paxos highlighted a global-uniqueness constraint risk for ledgers—Postgres only enforces uniqueness across partitions if the unique constraint includes the partition key, a crucial requirement to prevent duplicate application of idempotency keys. (blockchain.news)