At about 23:30 UTC on January 31, 2017, a GitLab engineer stopped an rm -rf command one or two seconds after realizing it was running on db1.cluster.gitlab.com, the production primary, rather than db2, its secondary. Roughly 300 GB had already been removed. The secondary could not take over because its own data directory had just been erased during an attempt to repair replication.[1]

The command is the memorable part of the story. It is not the most useful part.

GitLab.com remained down for about 18 hours and permanently lost roughly six hours of database changes. The project estimated at least 5,000 projects, 5,000 comments, and 700 new user accounts were affected. Git repositories and wikis lived elsewhere and survived; issues, merge requests, snippets, accounts, and other PostgreSQL-backed work inside the lost window did not.[1][2]

This incident deserves to be read as a recovery-system failure with an operator error in the middle. Load, replication, runbooks, host identity, backup binaries, alert delivery, storage placement, and restore bandwidth each removed one layer of safety. A snapshot created for staging—manually and for an unrelated experiment—became the only usable route back. The postmortem’s durable lesson is that a backup is an end-to-end service, not a scheduled command.

Image context: GitLab cofounders Dmitriy Zaporozhets and Sid Sijbrandij were photographed in December 2016, shortly before the outage. The portrait does not show the engineer, database, or recovery. It places the incident inside a young open-source company whose public response would make both its infrastructure and its treatment of an operator unusually visible.[6]

An ordinary load problem consumed the escape route

The failure chain began several hours before the deletion. At about 17:20 UTC, an engineer took an LVM snapshot of production and loaded it into staging to test whether pgpool-II could spread queries across multiple PostgreSQL servers. The normal snapshot ran once every 24 hours, but this experiment needed fresher data. That manual copy would later reduce permanent loss from almost a day to about six hours.[1]

Around 19:00, GitLab.com’s database load climbed. Spam contributed, but GitLab later found another expensive job in the mix: its abuse process had mistakenly scheduled a GitLab employee and associated data for removal after a troll reported the account. Comments on issues and merge requests began failing while engineers worked the load down.[1]

By about 23:00, the hot standby had fallen so far behind that the primary had already removed write-ahead log segments the secondary still needed. GitLab was not archiving those WAL segments, so the standby could not simply catch up. It had to be rebuilt with pg_basebackup, a procedure that began by emptying the secondary’s data directory.[1]

The repair then generated its own uncertainty. pg_basebackup appeared to hang without meaningful progress. Engineers raised max_wal_senders from 3 to 32; PostgreSQL would not restart because max_connections had been left at 8,000, so that setting was cut to 2,000. An strace showed the backup process waiting in poll but did not explain why. The postmortem later recorded that this silent wait could be normal while the primary prepared to send data, a behavior missing from GitLab’s runbooks at the time.[1]

This sequence matters because the engineer did not arrive at a shell prompt intending to touch production. Several hours of load response had turned into a manual replication rebuild, the secondary was already empty by design, and the next plausible troubleshooting step also involved an empty directory. Fatigue did not create the unsafe workflow. It made that workflow’s weak distinctions decisive.

db1 and db2 were too easy to confuse at the worst moment

At about 23:30, the engineer believed a previous pg_basebackup attempt might have placed files back in the secondary’s data directory. The directory needed to be empty before another attempt. The destructive command, however, ran against the primary.[1]

Independent reporting captured the immediate scale: only about 4.5 GB of the roughly 300 GB directory remained when the deletion stopped. It also reported GitLab’s candid live note that five backup or replication techniques were either unreliable or not properly configured.[4] The number made a dramatic headline, but the deeper defect was common authority. One operator, in a pressured repair session, could perform the same destructive step against either member of the database pair.

GitLab’s corrective work included changing shell prompts so hosts and environments were more visually distinct, improving the replication runbook, and automating recovery operations that had depended on manual commands.[1] Those controls address different failure moments. A louder prompt helps a human notice context before pressing Enter. A rehearsed runbook reduces improvisation. Automation can turn “empty the secondary and rebuild it” into an operation that checks role, hostname, replication state, and allowed target before it destroys anything.

None is sufficient alone. A red production prompt eventually becomes wallpaper. A correct document does not stop a pasted command. Automation can reproduce a wrong assumption at machine speed. The stronger design layers identity checks: production and staging use different access paths; primary and replica roles are queried rather than inferred from names; destructive tools demand a target-specific token; and routine resynchronization never grants an operator a generic deletion surface on both hosts.

Replication had copied the data, then copied the failure boundary

The hot standby was an availability mechanism. Under ordinary hardware failure, a usable secondary might have become primary. It was not an independent recovery copy for an incident in which the repair procedure deliberately erased it first.

That distinction is easy to lose because replication looks like continuous backup: current rows exist on another disk, on another host. Yet replication is optimized to converge. Deletions and corrupt updates can converge too, and administrative work on the replication pair can place both nodes in one change domain. In this incident, stale WAL forced a full resynchronization; wiping the secondary removed the failover path just before the primary was deleted.[1]

PostgreSQL 9.6’s own documentation described the missing complement. Archiving completed WAL segments makes it possible to start from a filesystem backup and replay changes to a chosen recovery point.[5] GitLab’s corrective list consequently included investigation of continuous archiving and point-in-time recovery, alongside rebuilding replicas. Replication and PITR answer different questions: Can another server serve now? and Can we reconstruct state from before a destructive event?

One architectural separation did work. Git repositories and wikis were stored outside the affected PostgreSQL database. They were unavailable while GitLab.com was down, but their contents were not part of the database loss. Developers with local clones could continue editing code even though hosted collaboration stalled.[1] The boundary preserved Git objects; it did not preserve the issues, comments, merge requests, snippets, accounts, and webhooks that gave those objects shared context.

That is why “the code survived” understates the impact. In an open-source project, a review explanation, issue history, or new contributor account can be the only durable record of why code should change. Separating blob stores limits blast radius, but each datastore still needs a recovery plan tied to the user work it uniquely owns.

Four documented procedures yielded one usable copy

GitLab’s final postmortem enumerated four recovery mechanisms: daily pg_dump backups uploaded to Amazon S3, daily LVM snapshots copied into staging, Azure disk snapshots used on some servers, and PostgreSQL replication for failover. At the point of recovery, only the staging copy made from that afternoon’s manual LVM snapshot was useful.[1]

The logical backup path had been failing before the incident. A regular application server ran pg_dump; because that host lacked the database cluster’s PG_VERSION file, GitLab’s Omnibus package selected PostgreSQL 9.2 tools against a PostgreSQL 9.6 database. The major-version mismatch made pg_dump exit. Cron sent failure messages, but the receiving system rejected them because those automated emails were not DMARC-authenticated. The S3 bucket was empty, and nobody knew.[1]

Azure disk snapshots existed for NFS servers but had not been enabled on the database hosts. The team assumed its other database procedures were enough and knew cross-storage-account restores could take hours or days. LVM snapshots did exist, but their purpose was refreshing staging rather than disaster recovery. One was almost 24 hours old; the other survived only because the engineer’s test had created it six hours before the deletion.[1]

The postmortem’s root-cause analysis asked why backups had not been tested regularly and answered plainly: no one owned that test.[1] That sentence connects the empty bucket, rejected email, version mismatch, and unmeasured restore path. Each component had a configuration. The recovery service had no operator responsible for proving that a production write could travel all the way through capture, storage, detection, restoration, and application validation.

Monitoring a successful process exit would still have been incomplete. A valid pg_dump can be truncated in transit, encrypted with a missing key, retained for too little time, or impossible to load into the available PostgreSQL version. The evidence that matters is a scheduled restore into an isolated environment, followed by application-level checks for recent rows, repositories, webhooks, permissions, and any external objects referenced by the database.

Restore throughput set the outage clock

The six-hour-old snapshot solved the recovery-point problem as well as the available evidence allowed. It did not solve the recovery-time problem.

GitLab had to copy the staging database back to production from slow Azure network disks. The postmortem put the effective storage ceiling near 60 Mbps and said the copy took around 18 hours; processor and network capacity were not the bottlenecks. At 17:00 UTC on February 1, the database returned without webhooks. Engineers restored those separately from another copy of the snapshot and completed final checks around 18:00.[1]

This turns two familiar acronyms into concrete loss. The recovery point was about six hours old, so database work after 17:20 UTC could not be reconstructed. The recovery time was about 18 hours, so even preserved Git repositories remained inaccessible through GitLab.com while bytes crossed slow storage.[1] Snapshot frequency constrains how much history disappears; restore bandwidth constrains how long surviving history remains unusable.

A credible restore drill must measure both. Copy a production-shaped dataset, not a convenient sample. Time snapshot discovery, permission changes, transfer, database startup, WAL replay, secret retrieval, application migration, and user-facing validation. Confirm that emergency compute can actually attach the storage where backups live. If the plan says “restore 300 GB” but has never moved 300 GB through the recovery path, its time objective is a hope with units attached.

Transparency should expose the system without sacrificing the operator

GitLab kept a public working document during the outage, posted status updates, and streamed the recovery. The final postmortem says the stream peaked near 5,000 viewers. That openness let users see scope and progress while the service itself was unavailable, and independent reports could distinguish the destroyed database from unaffected repositories and self-managed installations.[1][4]

The live document initially included the engineer’s name. GitLab later said it would redact names in comparable cases because another person might not be comfortable with that exposure.[1] In a follow-up devoted to “team-member-1,” the company shifted attention from individual blame to the system, confirmed the engineer was well, and said a promotion approved before the incident would still proceed.[3]

That response was not leniency in place of engineering. GitLab published the command path, configuration mistakes, missing ownership, storage constraints, and corrective issues in uncommon detail. Protecting the person made that technical candor more sustainable. If operators expect a single error to end a career, they have a reason to delay disclosure, hide near misses, and avoid taking the recovery work that reveals weak systems.

Blamelessness does not mean consequence-free operations. It means corrective action attaches to the conditions that made one mistake catastrophic: indistinguishable hosts, manual destructive repair, missing WAL archives, unobserved backups, untested restores, and no owner. Accountability becomes the obligation to change those conditions and verify the change.

The smallest credible recovery system

A small team does not need GitLab.com’s 2017 scale to inherit the same failure shape. One primary, one streaming replica, one nightly dump, and one object-storage bucket can create four reassuring nouns while still providing no tested recovery.

The minimum credible system is narrower. Give backup and restore a named owner. Run a version-matched logical or physical backup to storage outside the database host and its normal administrative domain. Keep WAL long enough to meet a declared recovery-point objective. Send failures through an observed channel independent of the job being monitored. Restore automatically on a schedule, validate application data, and record the age and duration of the successful test.

Then make the operational path harder to confuse. Separate production credentials and terminals, expose database role in the prompt, wrap replica rebuilds in checked automation, and require a second person for destructive production work when the incident is already long or the team is fatigued. Preserve a status channel and runbooks outside the service they describe. Benchmark the full restore using the volume and storage class production will actually require.

Teams that cannot staff those routines should simplify the promise: accept a longer documented recovery point, use a managed database with verified restore behavior, or reduce the amount of authoritative state. What they should not do is count replication and untested jobs as independent recoveries.

GitLab’s outage is remembered because one command erased a database in seconds. Its real warning matured over the next 18 hours. The command revealed how many mechanisms had never completed the last step: returning known-good data to a working service. The accidental staging snapshot carried the recovery because it alone had crossed enough of that path before anyone needed it.

Sources

1. GitLab, “Postmortem of database outage of January 31,” February 10, 2017 — final timeline, database topology, broken recovery paths, restore performance, impact, root-cause analysis, and corrective actions.
2. GitLab, “GitLab.com database incident,” February 1, 2017 — contemporaneous incident notes, affected data boundary, initial recovery state, and the six-hour staging copy.
3. Marcia Ramos, GitLab, “How is team-member-1 doing?”, March 17, 2017 — the project’s follow-up on operator welfare, transparency, learning, and the decision not to reverse a previously approved promotion.
4. Simon Sharwood, The Register, “GitLab.com melts down after wrong directory deleted, backups fail,” February 1, 2017 — independent contemporaneous reporting on deletion scope, backup state, and the usable staging snapshot.
5. PostgreSQL 9.6 documentation, “Continuous Archiving and Point-in-Time Recovery” — the historical WAL-archive and replay model relevant to GitLab’s missing recovery layer.
6. GitLab, “Gitlab-founders,” Wikimedia Commons — official photograph of cofounders Dmitriy Zaporozhets and Sid Sijbrandij, December 27, 2016.