The 2011 kernel.org break-in exposed three meanings of ‘trusted source’

The second-day group portrait from the 2011 Kernel Summit at LinuxCon Europe. The kernel.org compromise had opened the summit's technical agenda; this photograph documents the community conducting the recovery in public, not the intrusion itself.[1][9]

When administrators discovered on August 28, 2011 that kernel.org had been compromised, they lost the right to trust the machines that distributed the Linux kernel. They did not lose the ability to test whether the established Git history had changed. After an extensive investigation, the project reported no repository tampering—but it still rebuilt the infrastructure from the ground up, replaced credentials, retired its archive-signing key and kept most services offline for weeks.[1][2][7]

That combination is the useful part of the incident. “The source was intact” and “the source host was rooted” were both true. Git made one kind of integrity independently checkable; it did not certify the server process, the people operating a ref, the key signing a tarball or a developer endpoint caught in the same credential chain.

The postmortem therefore has three different meanings of trusted source:

Object integrity: these exact blobs, trees and commits are the history whose identifiers we already know.
Release authority: a recognized maintainer intended to publish this exact history or archive.
Runtime integrity: the machine delivering it is not stealing credentials, changing responses or abusing adjacent services.

The first claim survived the break-in. The second had to be rebuilt. The third was gone the moment the hosts were known to be under hostile control.

Image context: LWN's group photograph was made at the October 2011 Kernel Summit in Prague, where the kernel.org report was the first item discussed on October 24. The crowded frame matters because recovery was not only a disk-reinstallation job. It required maintainers to re-establish identities, keys, hosting boundaries and working agreements across a distributed community.[1][9]

A distribution point had quietly become a control plane

Kernel.org began as a place to distribute kernel releases. By 2011 it also hosted Git repositories, Bugzilla, Patchwork, wikis, kerneloops.org, email services and a customized gitweb layer. The transition happened incrementally: each service was useful, but together they turned a publication site into a dense identity and collaboration hub.[1]

The attackers appear not to have selected the Linux project for a source-code operation. The Kernel Summit report described a broader credential-stealing network that had already occupied many machines. It installed trojanized ssh and sshd binaries, watched terminal sessions, recorded usernames and passwords, and seemingly used forwarded SSH agents to reach more systems. The attackers operated quietly; investigators found no evidence that they altered kernel.org's data.[1]

That last sentence is a forensic finding, not an architectural pardon. Any password typed on an affected host, any private key stored there and any credential exposed through the compromised path had to be treated as lost. A clean repository comparison could not make a captured password secret again. Nor could it demonstrate that a live SSH daemon was the binary its package database claimed it to be.

The outage made the accumulated coupling visible. Kernel.org went offline, kernel development slowed, and the expected Linux 3.1 release remained pending. A minimal site returned on October 3 with only the mainline, stable and linux-next Git trees; other maintainers first had to inspect their own systems, generate new keys and re-establish access.[2] Git was distributed, so development did not disappear. The shared coordination point was important enough that its absence still changed the release rhythm.

Git gave responders an invariant outside the server

Git stores content as an object graph. A blob is addressed by a hash of its content and header. A tree points to named blobs and subtrees. A commit points to a top-level tree and to its parent commit or commits. Alter an old source file and its blob identifier changes; the tree containing it changes; the commit containing that tree changes; every descendant that names the altered commit changes as well.[3][5]

That structure made silent rewriting testable against information already held elsewhere. Kernel developers had complete clones on independent machines. They knew established commit identifiers and pulled often. A kernel.org repository that presented a changed object under an expected history would disagree with those copies. The Linux Foundation's contemporary explanation stressed both halves of the defense: content addressing made a changed history acquire new identifiers, while the large population of independent repositories supplied comparison points that the compromised server did not control.[3]

git fsck --full expresses the local version of that check. It verifies object hashes, object sanity, reachability and graph connectivity across loose and packed objects. A hash mismatch is a serious integrity failure; a missing object exposes a broken graph.[6] During recovery, those properties help answer: Does this repository contain the known history, intact?

They do not answer: Should this branch tip be trusted? A server administrator can point a ref at a new, internally valid malicious commit. Git can faithfully verify every object in that history without knowing whether Linus Torvalds or an intruder authorized the ref move. A hostile Git service could also exploit a vulnerable client or show different ref advertisements to different visitors. Content addressing constrains forgery of a known history; it does not turn an untrusted runtime into a harmless file server.

This boundary is why “Git was not compromised” must be read narrowly. The investigation found no data tampering, and the distributed graph supplied strong evidence for that conclusion.[1][3] It did not mean every service on the hosts was safe, every downloadable archive was authenticated, or every credential used against the machines remained trustworthy.

The archive signature had the wrong principal

Before September 2011, kernel.org automatically signed releases with one Linux Kernel Archives key. The current signature documentation records the uncomfortable recovery judgment: because of the systems compromise, that key was retired and revoked on December 11, 2011, and should no longer be used. Kernel.org says it is almost certain that the private key fell into malicious hands.[7]

The key's old placement made its statement weaker than many users assumed. An automatic signature proved that an archive had passed through kernel.org's signing process. If the distribution host and its signing credential were compromised together, the signature did not independently attest that a maintainer intended those bytes to be a release. It repeated the host's claim in cryptographic form.

Kernel.org later cross-checked older releases, signed them with a one-off key created for that task, and destroyed the private half so it could not bless any future artifact.[7] Its current workflow separates two claims. A dedicated off-network system signs checksums that help users compare mirror contents with central storage, but the documentation explicitly says those checksums do not replace developer signatures. A maintainer's signature supplies the authorizing statement that mirror consistency cannot.[7]

This is a deeper recovery than changing algorithms or generating a longer key. The principal changed. “Kernel.org served this” stopped masquerading as “a kernel maintainer released this.” Transport and authorization became separate evidence.

The rebuild reduced ambient authority

On September 23, H. Peter Anvin described the replacement infrastructure in a message to the kernel mailing list. Developers would no longer receive shell access to Git repositories. Gitolite would mediate pushes using new SSH credentials, and additional services would return only as resources allowed and as they could be operated securely.[4]

The Kernel Summit report added the organizational boundary. Services would be separated more aggressively; operations would move fully into the Linux Foundation; additional staff would support the work; and only full-time administrators would retain root access. Mailing lists that already worked elsewhere would stay elsewhere rather than be folded into the rebuilt host.[1]

Release upload changed too. The kup tool accepts software for kernel.org only after cryptographic verification against a list of pre-approved maintainers. Its documented path expects a signature over the uncompressed .tar, verifies it, and then handles the compressed publication formats. The automated route begins with a signed Git tag and identifies both the source tree and the maintainers allowed to publish from it.[8]

None of these controls is magical. Gitolite can have vulnerabilities. A maintainer key can be stolen. Full-time staff can make mistakes. The improvement is compositional: a web application no longer needs general shell authority over repositories; a distribution host no longer invents release authority by signing whatever reaches it; and one machine no longer has to contain every collaboration service. A failure has fewer automatic paths into the next trust domain.

Recovery needed four proofs, not one clean clone

The incident suggests a compact recovery model for any project running its own forge or release host.

State proof establishes the intended repository graph. Compare full object identifiers and refs with known independent clones, signed tags, release manifests or another external record. A backup alone is not enough if it may have captured the attacker's state.

Authority proof establishes who intended to advance that graph. Verify a signed tag or manifest against a key whose fingerprint was learned outside the compromised service. If that key ever resided on or transited the host, rotate it and publish the revocation through independent channels.

Runtime proof establishes a trustworthy execution base. Reinstall or replace affected hosts from known media, rebuild configuration deliberately, and issue new service secrets. A passing git fsck says nothing about a trojanized SSH binary, a malicious web hook or a stolen CI token.

Identity proof closes the credential lineage. Inventory passwords entered, keys stored, agents forwarded, tokens readable and administrator sessions active during the exposure window. Revoke first; reissue through a channel that does not depend on the compromised identity. The 2011 rebuild's new SSH credentials and reduced shell access followed exactly this boundary.[1][4]

For a two-maintainer library, this does not require a second datacenter or a custom upload service. It can mean an off-forge read-only mirror, release tags signed with a hardware-backed or offline key, separate credentials for CI and publishing, and a recovery record containing known-good refs and key fingerprints. The maintainers must be able to operate those controls; an elaborate system nobody rehearses is weaker than a small one they can restore.

For a large foundation or distribution, the bar rises. Preserve audit logs outside the service that generates them. Isolate release signing from general-purpose web hosts. Restrict SSH agent forwarding. Give dynamic services separate identities and failure domains. Rehearse a return with only the essential repository path before restoring convenience layers. Kernel.org's limited October return is a useful model: recovery order should follow authority, not homepage completeness.[2][4]

The falsifying test is simple. If compromising one administrator workstation or one public host still lets an attacker change refs, issue a trusted release, erase the audit trail and steal every recovery credential, the architecture has only renamed its single point of trust.

What survived was the evidence, not the machine

The 2011 compromise is often remembered as an example of Git's tamper evidence, and it is one. But Git's success did not shrink the incident to a non-event. The hosts were rebuilt, the archive key was revoked, credentials were replaced, services were separated and publication authority was pushed back toward individual maintainers.[1][4][7][8]

The durable lesson is not that distributed version control makes hosting security optional. It is that a resilient project keeps enough evidence outside any one host to say exactly what survived when that host becomes untrustworthy.

A trusted source is therefore not one green check. It is a chain of distinct claims: these are the expected bytes; this maintainer authorized them; this delivery system has not silently substituted something else. Kernel.org could prove the first claim while rebuilding the other two. That separation turned a rooted server from an unknowable source-code catastrophe into a bounded—but still expensive—recovery.

cronfeed.work