Gentoo's GitHub mirror was compromised. Its source of truth was not.

GerardM photographed Gentoo developer Patrick Lauer, left, with Jimmy Wales at FOSDEM 2005. The scene predates the 2018 breach, but it shows the contributor-facing open-source community that Gentoo's GitHub presence later helped serve; the photograph is context, not incident evidence.[8]

At 20:19 UTC on June 28, 2018, an attacker successfully entered an administrator's GitHub account. At 20:25, the attacker created another administrator and began ejecting Gentoo developers from their own organization. By 20:38, a destructive command had appeared in the public gentoo/gentoo mirror. GitHub froze the organization at 21:28.[1]

The takeover was real, the repositories visible on GitHub were untrustworthy, and some contribution infrastructure would remain unavailable for about five days.[1][2] Yet Gentoo's canonical package repository on project-controlled infrastructure was not altered. The project had not made its most popular public copy its ultimate authority, so responders possessed something more useful than a backup: an independent record of what the branches were supposed to contain.[1][2] Contemporaneous independent Linux reporting drew the same distinction between the compromised GitHub copy and Gentoo's unaffected master ebuild repository.[3]

That architectural boundary is why the incident deserves a careful reading. “The mirror was not canonical” did not make the breach harmless. It constrained one part of the blast radius while exposing how much state—administrator access, pull requests, CI, repository settings, and contributor trust—lived outside the Git object database.

Image context: the FOSDEM photograph was made in February 2005, long before the compromise. Patrick Lauer's Gentoo shirt places the project inside the face-to-face contributor culture that public forges extend online. The image does not show the attacker or the 2018 response.[8]

Seventy minutes from guessed password to containment

Gentoo's postmortem traced the root cause to a password used by an organization administrator. The collected evidence suggested a pattern in which disclosure on one site made passwords on unrelated sites guessable. One successful login therefore arrived with organization-owner authority: the attacker could invite new accounts, grant them administrative access, remove legitimate members, change the billing email, and rewrite repositories.[1]

The sequence was loud. Six minutes after the successful login, a dummy administrator was added and valid users began disappearing. Removal emails alerted developers; the first person noticed trouble at 20:29, the infrastructure team received a report at 20:36, and the first GitHub abuse report went out at 20:49. GitHub's freeze at 21:28 put the interval from account entry to platform containment at roughly 69 minutes.[1]

That quick detection was partly luck, as Gentoo later acknowledged. A quieter intruder could have left membership intact and changed less conspicuous content. Instead, the attacker force-pushed rewritten histories, defaced a README, and placed deletion commands in build-related files. The same force-pushes that destroyed trusted branch tips also made existing checkouts less likely to accept the new histories silently through an ordinary pull.[1]

Incident speed should therefore be read in two columns. The response machinery was fast: developers noticed, escalated through formal and personal channels, and found the compromised account using audit logs. The adversary was also noisy. An organization should preserve the first advantage without assuming the second.

Three repositories were poisoned, but not in the same way

Gentoo identified malicious-content windows for three repositories. Initial clones of gentoo/gentoo between 20:38 on June 28 and 06:58 on June 29 could contain the altered tree. The corresponding windows were 20:56–06:59 for gentoo/musl and 21:07–06:57 for gentoo/systemd. Gentoo advised anyone who cloned during those intervals to discard the clone and start again from a repaired source.[1]

The payloads were crude but the distribution boundary matters. The attacker inserted variants of rm -rf into ebuild or configuration content. Gentoo judged those commands unlikely to execute for end users because of technical guards, and its report did not claim that the public compromise had reached the project-hosted repositories.[1] That is narrower than saying “nothing happened.” A fresh clone could still hold attacker-controlled content, and responders still had to treat every GitHub branch tip as hostile until it was compared with an independent authority.

The repositories also had different provenance. Gentoo's main repositories lived on Gentoo infrastructure and were mirrored to GitHub to meet contributors where they already worked. The incident report singled out gentoo/systemd as an exception: it was stored directly on GitHub rather than mirrored from Gentoo. That exception complicated recovery because “copy the known-good canonical branch back over the mirror” was not a complete answer for every repository.[1]

This is the first durable lesson. Labels such as mirror, upstream, and canonical must describe an enforced dataflow, not team folklore. For each repository, operators need to know which system may advance authoritative refs, which systems only replicate them, and where an independent known-good tip can be found when the forge's current state becomes evidence under investigation.

The source survived; the collaboration surface did not

While GitHub was unavailable, Gentoo development outside the Proxy Maintainers workflow continued, and the postmortem recorded more than 700 commits during the incident. That continuity is strong evidence that the canonical-repository split worked.[1] It is not evidence that GitHub was disposable.

Pull-request CI went down. Many outside contributors depended on GitHub pull requests through the Proxy Maintainers Project. Past pull requests became disconnected from their original commits and were closed; GitHub could not reconnect them, so contributors had to open new ones. The code objects could be restored, but the relationships around those objects—review conversation, authorship workflow, CI state, and open contribution queues—did not simply reappear when a clean branch was force-pushed.[1]

Gentoo also discovered that it lacked a backup of the GitHub organization's configuration. Its action list called for offline copies of organization settings, external capture of audit events, fewer owners, stronger retirement procedures, required two-factor authentication, and mirrors for repositories that existed only on GitHub.[1] In other words, the postmortem widened “repository recovery” into control-plane recovery.

Mozilla's published repository-security guidelines draw a compatible boundary. They recommend requiring two-factor authentication, limiting production-branch access, applying branch protection to administrators, signing production commits and important tags, and auditing the commits that compose a release. They also call for regular review of organization and repository settings, treating forge configuration as state to inspect rather than a promise to assume.[4]

A Git hash proves content, not permission

Git's content addressing made comparison precise. Gentoo's postmortem recorded known legitimate and malicious commit IDs, then restored the three affected repositories to good tips between 06:57 and 06:59 on June 29.[1] A responder could say exactly which tree a branch referenced before and after repair.

But a commit hash does not answer the authorization question. If a compromised administrator moves main to a new hash, Git can verify that the downloaded objects match that hash; it cannot infer that the project intended the move. The missing claim is not “are these bytes internally consistent?” but “did an authority we recognize approve this ref?” Signed release tags can supply that extra statement when consumers verify them against keys learned outside the compromised forge; git verify-tag exists specifically to validate those signatures.[7]

Signatures are not a complete incident plan. A signing key can itself be stolen, a project can sign too many routine events for verification to mean much, and contributors can ignore failed checks. Their value comes from separating authorities: forge login controls who may operate the platform, while a distinct signing key attests to a release or ref. If both factors live in the same browser session or automation secret store, the separation is cosmetic.

Current GitHub controls can narrow the platform side of this problem. Organizations can require two-factor authentication for members, outside collaborators, and billing managers.[5] Protected branches can block force-pushes, restrict pushers, require reviews or signed commits, and apply rules to administrators—but the documentation also notes that administrators and bypass roles are exempt by default unless the project explicitly closes that path.[6] A control that the compromised role can silently bypass is weaker than its settings-page label suggests.

Recovery required two independent proofs

The repository proof was a comparison: known Gentoo-hosted refs versus the altered GitHub refs, followed by a controlled restoration. The account proof was an audit: identify the entry account, remove its access from Gentoo systems, rotate credentials, inspect logs, and prevent the stolen password from working again. By 23:40 on June 28, Gentoo had identified the entry point and removed its access from primary project properties. The repaired repositories were pushed the next morning, but GitHub did not make the organization public again until 11:46 on July 3.[1]

That gap explains why “we have a clean clone” is not a closure criterion. Responders also needed to understand organization membership, the malicious accounts, changed email settings, pull-request damage, and the activity of the compromised user beyond GitHub. Code restoration answered what should be served. Control-plane review answered who could change it again.

For a small project whose forge is the only canonical host, the Gentoo design cannot be copied literally during an incident; there is no second authoritative service waiting behind the mirror. The practical substitute is an off-platform recovery record: frequently updated read-only clones, protected and externally verified release tags, exported organization configuration, an audit stream outside the forge, and at least two reachable owners using phishing-resistant authentication. None of those turns a copy into authority by itself. Together they preserve the evidence needed to rebuild authority deliberately.

For a larger project with its own Git service, calling GitHub a mirror is still insufficient. Test the direction of replication. Inventory GitHub-only exceptions. Confirm that a compromised mirror cannot write back to canonical refs. Rehearse the commands and communications needed to freeze contributions, publish known-good commit IDs through an independent channel, rebuild the mirror, reissue CI credentials, and reopen only after membership and settings are audited.

Gentoo contained the dangerous code because it knew where truth lived. Its harder losses appeared where truth had never been reduced to Git objects: identities, permissions, reviews, and contributor workflow. The lasting postmortem lesson is not “self-host everything” or “mirrors make breaches safe.” It is to separate the system that distributes convenient copies from the evidence that tells people which copy the project actually authorized.

cronfeed.work