Apache's 2010 attackers turned a password reset into root

Six members of the Apache Infrastructure team show their matching shirts at ApacheCon US 2009 in Oakland, five months before the attack. The photograph is not of the incident response; it shows the human operations team whose accounts and services were targeted. Photograph by Noirin Shirley.[6]

The decisive step in the Apache Software Foundation's April 2010 infrastructure breach came after its report could no longer distinguish whether a JIRA cross-site scripting attack or concurrent password guessing had yielded the administrator foothold. It was a password reset that looked routine.[1]

By then, the attackers had already obtained JIRA administrator rights, converted an attachment setting into server-side code execution, and modified the login path to record passwords. They used JIRA itself to send reset messages to infrastructure administrators. When one administrator returned and restored a familiar password, the planted recorder captured a secret that also unlocked a local shell account with full sudo access. A web-application compromise became root on brutus.apache.org without another privilege-escalation vulnerability.[1]

That chain makes the incident worth reconstructing. It was neither a single clever bug nor proof that every Apache system had fallen. Five boundaries failed in sequence; other boundaries held. The issue-tracking host was no longer trustworthy, three application databases had to be treated as copied, and cached Subversion credentials carried the attackers onto a second machine. Yet they did not elevate there, LDAP was largely unaffected, and Apache's source-code repository was not compromised.[1][4]

The photograph above was taken five months before the attack.[6] Its line of infrastructure-team shirts is part of the security model: a volunteer foundation runs through people moving among conferences, browsers, ticket queues, shell accounts, and shared services. The 2010 lesson is not “do not click links.” It is that no ordinary action in that human system—a ticket view, an attachment upload, a password reset, or a cached login—should silently inherit the authority of the next layer.

A ticket became executable code

On April 5, an attacker using a compromised Slicehost server opened INFRA-2591 in Apache's JIRA. The issue text included a shortened URL presented as an error report. It redirected back into the same JIRA instance with a cross-site scripting payload designed to steal the session cookie of whoever opened it. Several members of the infrastructure team followed the link, exposing sessions that included JIRA administrator privileges. At the same time, the attackers aimed hundreds of thousands of password guesses at login.jsp.[1]

By April 6, one route had succeeded; Apache could not tell which of the two methods supplied that administrator foothold. The attackers used JIRA administration to disable notifications for a project and change the directory used for uploaded attachments. They selected a location that was both writable by the JIRA process and configured to execute JSP files. New issues then supplied the uploads: one JSP browsed and copied files; others preserved backdoor access under the account running JIRA.[1]

The XSS flaw mattered, but it did not contain the whole failure. Stolen application administration became code execution because a setting could redirect untrusted attachments into an executable path. The resulting process access was unusually broad because the JIRA daemon ran as the role account that had installed the application. A browser session, application configuration, writable storage, executable content, and filesystem identity formed one continuous authority path.

Atlassian's response shows why “patch the XSS” was not yet a complete closure statement. Its April 16 advisory described XSS and privilege-escalation vulnerabilities capable of reaching the filesystem, and the final patch superseded two earlier fixes referenced by Apache's initial report. The advisory covered JIRA 3.13.4 through 4.1 and told customers to apply the replacement patch even if they had installed the first ones.[2] That is a narrow historical fact, not a claim about current JIRA. It is also a durable incident rule: an emergency fix has a version and a scope; it is not evidence that every route discovered during response is closed.

The login page became a credential bridge

By the morning of April 9, the attackers had installed a JAR that recorded passwords entered at login. They then triggered password resets for members of Apache's infrastructure team. The messages arrived through a service the recipients already administered, and changing a temporary password back to the usual one looked like recovery from an application glitch.[1]

One restored password was also the password for a local account on brutus, and that account had full sudo. Password-based SSH access was still possible. Apache's team had tried to disable it, but UsePAM yes left an authentication route open through PAM. Optional OPIE one-time passwords existed on the host, yet they were not mandatory for every superuser. The attackers took the password they had captured inside JIRA, logged into the operating system, and became root.[1]

This is where a shallow postmortem can choose the wrong unit of analysis. “Password reuse” is true but incomplete. Reuse only became catastrophic because four conditions aligned: the compromised application could record plaintext credentials; the same secret represented both application and host identity; the host still accepted that kind of secret remotely; and the account could cross directly into root. Removing any one edge would have changed the outcome.

Nor would multifactor authentication on the original JIRA session alone have broken every path. The attackers were already inside the application when they altered its login behavior. A modern control set must protect the administrator session, separate application and host identities, prevent the application from becoming a credential-collection surface, and require a separately authenticated elevation step at the operating-system boundary. Controls overlap because the attack moved.

Root stopped at one host, but secrets did not

Root on brutus meant Apache could no longer trust the machine. It hosted JIRA, Confluence, and Bugzilla, so the response presumed that all three databases had been copied. JIRA and Confluence stored unsalted SHA-512 password hashes; Bugzilla used salted SHA-256 hashes. Apache advised users to retire the passwords in either case and treated anyone who had logged into JIRA from April 6 through April 9 as having exposed the password itself to the modified login form.[1]

The attackers also found cached Subversion authentication credentials in user home directories. Those passwords let them sign in to minotaur.apache.org, also known as people.apache.org, Apache's main shell server. This was lateral movement, but not total takeover: the compromised accounts did not yield privilege escalation on minotaur. Other hosts and services, including LDAP, were largely unaffected. Apache's infrastructure vice president separately told Computerworld that the source repository had not been compromised and that no source code was affected.[1][4]

Service isolation therefore worked in the only honest sense: partially. It could not protect databases co-resident on the rooted host, and cached credentials made host boundaries porous. It did prevent one root compromise from automatically becoming foundation-wide administrative authority. The useful question is not whether segmentation “worked,” as if it were binary. It is which identities, secrets, data stores, and elevation paths remained valid after one node was lost.

Apache had received a related warning in August 2009. A compromised ApacheCon host exposed a backup account's SSH key, which attackers used to reach an unprivileged account on minotaur. They planted CGI scripts in website document roots, and an automated rsync moved those files to a production web server. That earlier report said Subversion data lived elsewhere and source code was never at risk.[3] The two incidents were different, but both demonstrated the same structural hazard: credentials and automation can carry authority across a host boundary even when the first account is unprivileged.

Recovery meant replacing trust, not cleaning files

Apache detected the 2010 attackers about six hours after the password resets began and started shutting services down. Because root access on brutus had lasted for hours, searching for known malicious files could not re-establish trust in its operating system. The team moved services to another machine, thor.apache.org. JIRA and Bugzilla were back by April 10; Confluence remained offline when the incident report was published on April 13.[1]

The remediation list followed the failed edges. JIRA was reinstalled by root but ran as a separate, restricted daemon identity. Temporary reverse-proxy exclusions blocked risky picker paths while the application fix matured. One-time passwords became mandatory for superusers. Apache disabled cached SVN passwords globally with store-passwords = no under the Subversion [auth] configuration and removed existing caches. Fail2Ban coverage was extended to JIRA login failures.[1]

Each measure had a specific job. Separating installer and runtime identities reduced filesystem authority. Rebuilding on another host answered the loss of root trust. One-time credentials and SSH changes narrowed reusable host secrets. Disabling SVN password storage removed a lateral-movement cache. Rate limiting addressed the concurrent guessing campaign. None of them, alone, could be labeled “the fix.”

The public report was itself part of recovery. It named the host roles, dates, password-hash properties, successful and failed escalation points, configuration mistakes, and changes still in progress. Bruce Schneier highlighted it at the time as a model of transparent incident reporting.[5] That praise should not turn disclosure into theater: the report was useful because another operator could map its facts onto concrete controls, not because it merely admitted a breach.

Read the incident as an authority graph

For maintainers, the cleanest modern exercise is to redraw this attack without the 2010 product names:

untrusted ticket → privileged browser session → application administration → executable storage → runtime identity → captured login secret → host elevation → cached cross-host credential

Then require evidence that every arrow has a break. Can issue content execute in an administrator's origin? Can an application administrator choose an executable upload destination? Does the service identity own its installation? Can a recovery workflow collect a reusable host secret? Does remote login accept that secret? Can one account elevate without a separate factor? What credentials remain readable after the host is rooted, and what second systems accept them?

That graph also preserves the incident's boundary. Apache did not report altered source code, and independent contemporary reporting confirmed that the repository remained outside the compromise.[4] The right conclusion is not that separation failed because attackers reached two machines, or that it succeeded because releases stayed clean. The issue host was lost, user credentials were exposed, lateral movement occurred, and other authorities remained intact—all at once.

Apache's attackers reached root through a password reset because the reset crossed layers that had been designed as conveniences rather than security boundaries. The lasting postmortem is to make those layers visible. Patching the first flaw closes an entry point. Breaking the authority chain decides how far the next one can travel.

cronfeed.work