Matching kernel oopses exposed Debian’s four-server breach

Debian contributors at DebConf3 in Oslo in July 2003, four months before the compromise. Lars Wirzenius's group portrait documents the human trust network behind the project's distributed infrastructure.[8]

On the evening of November 20, 2003, repeated Linux kernel failures on Debian's master server looked like a machine problem. Administrators then discovered that murphy had been producing the same kind of oops—the kernel's report that it had reached an invalid internal state. Two long-running systems failing alike made hardware a poor explanation. Filesystem monitoring was also reporting that /sbin/init had been replaced. By 22:00 UTC, administrators had confirmed that Debian infrastructure was under hostile control.[2][4]

The breach eventually covered four named hosts: klecker, master, murphy, and gluck. Together they carried the bug tracker, mailing lists, web and CVS services, developer pages, quality-assurance systems, and security-related services. The attacker obtained root on each machine and installed the SucKIT rootkit. Yet Debian's package archives were not altered.[1][2]

That is not a story about an attack that “failed.” Developer work stopped, credentials became suspect, machines were reinstalled, and parts of the installer project lost access to builds. It is a story about three different boundaries failing or holding independently:

A stolen contributor identity opened the first door.
A local kernel flaw turned an ordinary account into root.
Package integrity remained testable because crucial evidence existed outside the compromised hosts.

The DebConf3 portrait above was made four months earlier, when Lars Wirzenius gathered Debian contributors into one frame in Oslo.[8] It puts the incident's “accounts” in their proper context: each credential represented a person in a distributed community, but shared infrastructure made those individual trust relationships composable.

The useful postmortem question is therefore not simply How did four servers get rooted? It is Why did the breach spread as far as it did, how did operators notice it, and what let them make a defensible claim about the archive afterward?

One credential became a route through the project

Debian's reconstructed timeline begins at about 17:00 UTC on November 19. Someone logged into klecker with a previously sniffed password for an unprivileged developer account. Within minutes, the attacker fetched source code for a local exploit over HTTP, gained root, and installed SucKIT. Investigators later recovered a Burneye-obfuscated exploit executable during forensics. The same developer credentials also worked on master, where the compromise sequence repeated.[2]

murphy was more restricted: that developer could not log in directly. Root on master, however, exposed an administrative account used for backups. That service path provided access to murphy. On November 20, a password captured on master opened gluck. The attacker did not need one universal back door. Each successful host supplied the credential or authority needed for the next move.[2][4]

This chain matters because “developer access,” “root access,” and “service-to-service access” were not interchangeable powers, but they composed badly. The stolen password supplied presence. The kernel exploit supplied privilege. Backup access and password sniffing supplied lateral movement. Removing any one link would have changed the blast radius even if the other two still existed.

The original credential's precise theft path was never established publicly, and the attacker was not identified. Debian's reports are appropriately narrower: they describe a sniffed password, the logins observed, the exploit recovered, and the rootkit found. That boundary keeps the postmortem anchored in forensic evidence rather than retrofitting a complete attacker biography.[2][4]

The rootkit made the machines noisier than the attacker intended

SucKIT modified a running Linux kernel without relying on a loadable kernel module. It could hide files and processes and capture passwords; Debian's investigators connected it to replacement of /sbin/init and to the kernel oopses that first drew attention. AIDE, the file-integrity monitor installed on several hosts, also noticed the changed binary and suspicious timestamps under /usr/lib/locale/en_US.[2]

The decisive signal was correlation. A single oops on an old server can plausibly be hardware, a driver, or a one-off kernel defect. The same unfamiliar behavior on two systems within the same administrative estate changes the prior probability. Debian's administrators moved from maintenance to incident response when the failures stopped looking independent.

Monitoring alone did not produce that judgment. In the later LWN discussion, an administrator involved in discovery clarified that AIDE had reported changes, but the alert was initially assumed to reflect another administrator's legitimate work. The tool detected a fact; the team still needed an owner to close the loop.[7] This is the operational difference between alert delivery and alert resolution. An integrity warning with no explicit acknowledgment path can be accurate and still fail as a control.

The incident also warns against celebrating accidental detection. A defective rootkit exposed this attack by destabilizing more than one kernel. A quieter implant could have preserved the same credentials and authority without the oopses. The transferable control is not “attackers make mistakes.” It is to compare low-frequency anomalies across hosts and to require human closure for changes to high-authority files.

The kernel bug had been fixed, but not recognized as a security boundary

Forensics separated the privilege-escalation exploit from the rootkit installed afterward. The exploit abused an integer overflow around the kernel's brk handling to change memory protections and gain control of kernel memory. It became CAN-2003-0961, now written CVE-2003-0961. Debian's DSA-403 advisory shipped corrected kernels for its supported Linux 2.4 packages.[2][5]

The timing is uncomfortable. A code correction had entered a Linux 2.6 prerelease in late September and the 2.4 line in early October, but its security significance was not understood. No vendor advisory had told operators that a local account could turn this bug into root. Linux 2.4.23, released on November 28, contained the fix; Debian published DSA-403 on December 1.[2][5]

Calling the compromised hosts merely “unpatched” would erase the important part. A fix existing upstream is not the same as an identified vulnerability, a backported package, an operator notification, and a completed fleet rollout. In November 2003, that translation had not happened before exploitation. The incident sits in the gap between code maintenance and vulnerability response.

That gap remains relevant to projects that grant shell access. A local privilege boundary is only as strong as the oldest exploitable kernel beneath it, but patch urgency depends on knowing which ordinary fixes are security fixes. Shared development hosts therefore combine two hard requirements: timely kernel maintenance and an assumption that any contributor account may eventually be stolen.

“The archive was not affected” was a result, not a premise

Debian's first public notice on November 21 named the affected machines and immediately distinguished them from the package archive. It did not simply turn services back on. The security archive was held for verification, and the already-mirrored Debian 3.0r2 point release was checked before its postponed announcement proceeded.[1]

The final investigation report explains how that conclusion was reached. Responders compared archive state with several records that the compromised machines did not exclusively control:

lists of MD5 checksums accumulated on other systems;
digitally signed .changes files preserved in external mailing-list archives;
signed .changes files on the archive servers; and
mirror logs stored outside the affected hosts.[2]

None of those records was a magic seal. MD5 is no longer an appropriate collision-resistant integrity hash. A signature is only as trustworthy as its key custody. A mirror can faithfully preserve malicious content if it receives that content after compromise. The strength came from independent provenance and agreement: the attacker would have needed to rewrite multiple kinds of evidence in multiple places to make altered packages resemble the pre-incident archive.

This is different from restoring a backup. A backup answers, “Can we recover some earlier bytes?” The archive evidence answered, “Do the bytes under review agree with records created before and outside this incident?” Recovery needs both questions. A pristine copy without provenance can restore an attacker's state; provenance without a recoverable copy can prove loss without repairing it.

The archive's clean bill also had a scope boundary. It did not make the four hosts trustworthy again, rescue every service, or prove that every developer credential remained secret. Debian's installer team reported on December 12 that beta 1 images and some daily builds on an affected machine were still unavailable, even though known-good snapshots had surfaced elsewhere.[6] Package-archive integrity survived; operational continuity did not.

Recovery revoked the graph before rebuilding the nodes

Responders created images of compromised disks for analysis. master, murphy, and gluck were reinstalled, then services were returned one at a time. klecker was initially cleaned and hardened remotely so the security archive could resume sooner, with a full reinstall deferred. The timeline records security service returning early on November 22 and mailing lists on November 25.[2][4]

Host recovery was only half the work. Debian disabled developer accounts in LDAP, removed authorized SSH keys from important machines, invalidated passwords on related services, and replaced SSH host keys. Its operational notice asked developers to stop using Debian credentials until told otherwise and to change any password reused elsewhere.[2][3]

Those actions recognize a property that disk imaging cannot solve: once a password sniffer has run as root, the credential graph is contaminated. The correct unit of recovery is not the server. It is every identity, key, passphrase, backup account, and outbound login that crossed the server during the exposure window.

That widened response had a real cost. Uploads and CVS access stopped because the same identity system that enabled distributed contribution also connected contributors to shared production services. The project accepted the outage rather than treating convenience as evidence that credentials were probably safe.[3][4]

A current design test for shared OSS infrastructure

The 2003 incident predates today's hosted forges, hardware-backed credentials, short-lived workload tokens, and routine immutable infrastructure. Its control logic is still current.

For a small project using a managed forge, the lesson is not to recreate Debian's server estate. Keep release authority out of an everyday maintainer session. Give CI a narrowly scoped, short-lived publishing identity; preserve signed release manifests or tag fingerprints somewhere the forge account cannot rewrite; and retain a tested export of issues, repository refs, and package metadata. The boundary condition is operational capacity: controls nobody can restore under pressure are decorative.

For a distribution or foundation that must run shared systems, test four separations:

Contributor identity from host privilege. A stolen contributor credential should not become root because one local kernel bug exists.
One service from the next service. A backup account should not be a general lateral-movement credential, and secrets captured on one host should not authenticate broadly.
Detection from the system being detected. File-integrity results and audit events need an external destination, an accountable owner, and a deadline for explanation.
Release state from release evidence. At least one verification record—signed promotion metadata, transparency entries, independent mirrors, or known-good manifests—must live beyond the administration domain that can change the packages.

The falsifier is straightforward. If one stolen maintainer password plus one host privilege escalation can still change release artifacts, erase the only audit trail, obtain every recovery secret, and rewrite the only record used to declare the archive clean, then the project has reproduced the 2003 password chain with newer tooling.

Debian's breach became visible because two machines failed in the same strange way. Its most important integrity claim survived for the opposite reason: the records did not all fail together. Correlated anomalies exposed the shared compromise; deliberately uncorrelated evidence bounded what the compromise had reached.

That is the durable postmortem. Resilience did not mean keeping every service online or preventing root on every host. It meant preserving enough independent evidence to stop four rooted servers from becoming an unknowable package archive.

cronfeed.work