Prod DELETE disaster debate

A database can erase live customer records in one command, and one widely shared engineering thread centers on what happens after that command runs in production. (x.com) In the thread, a junior engineer runs an unfiltered `DELETE` query against production, and the team’s discussion turns to recovery steps and missing guardrails rather than a single person’s mistake. Hacker News has circulated the same failure pattern for years: developers with broad production access can wipe data quickly if controls are weak. (x.com) (news.ycombinator.com) The basic problem is simple: a hard delete removes rows immediately, while a soft delete marks them as inactive so they can be restored later. Google Cloud says soft delete retains deleted objects for a recovery window, and Microsoft says Azure Backup soft delete keeps deleted backup data for an added retention period of up to 180 days, with 14 days as the default. (cloud.google.com) (techcommunity.microsoft.com) Backups solve a different part of the problem: they recreate past state after damage is done, but they do not stop the bad command from running. Amazon Web Services says continuous backups can support point-in-time recovery, which restores a database to a specific moment using snapshots and transaction logs. (docs.aws.amazon.com 1) (docs.aws.amazon.com 2) The debate in the thread lands on change controls that slow down destructive actions before they reach production. GitHub says branch protection rules can require approving reviews and passing checks before merges, and CODEOWNERS can force review requests to designated owners for sensitive parts of a codebase. (docs.github.com 1) (docs.github.com 2) Those controls are common in source code, but database consoles and ad hoc scripts often sit outside the same approval path. The result is a split system in which application code may need two approvals, while a direct production query can still run with one credential and no second check. (docs.github.com) (news.ycombinator.com) Cloud vendors increasingly pitch soft delete as a way to preserve speed without making every cleanup task a high-friction ceremony. Google says soft delete can let teams move faster when pruning old data because there is an undo window if a deletion turns out to be a mistake. (cloud.google.com) That trade-off has limits. Soft delete adds storage cost and retention rules, and it fits object stores and backups more neatly than relational tables with foreign keys, legal deletion requirements, or workloads that expect rows to disappear immediately; Amazon and Microsoft both frame backup retention and restore as policy choices with operational constraints. (docs.aws.amazon.com) (docs.azure.cn) What the thread captures is a familiar engineering calculation: every safeguard added before production can slow routine work, and every safeguard skipped shifts the cost to recovery after a mistake. The command is short, the restore is not. (x.com) (docs.aws.amazon.com)