PHP’s backdoor commits bypassed gitolite through a second push path

William Stadtwald Demchick photographed PHP creator Rasmus Lerdorf at the New Zealand PHP Conference in 2014. The 2021 attacker later borrowed Lerdorf’s name as cosmetic authority; this portrait documents the person, not the origin of the malicious commit.[7]

On March 28, 2021, a commit called [skip-ci] Fix typo appeared in PHP’s source tree under the name of project creator Rasmus Lerdorf. The title promised housekeeping. The diff added 11 lines to ext/zlib/zlib.c. Those lines looked for a misspelled User-Agentt HTTP header, checked its value for the word zerodium, and evaluated the value from a fixed eight-byte offset—an arbitrary-code-execution backdoor in the interpreter itself.[2]

The code was noticed and reverted. Later that day, a second malicious commit restored it under the name of core developer Nikita Popov. That was reverted too. PHP’s official account says neither commit reached a release or an end user; the project then paused releases for two weeks while it investigated.[1][4]

That containment makes the episode easy to remember as a triumph of open review over a clumsy attacker. It was—but that is only half the postmortem. The more durable lesson arrived nine days later, when Popov explained why the project’s first forensic story had been wrong. The suspicious writes did not appear in the logs for PHP’s expected SSH-and-gitolite workflow because they had entered through a second, password-authenticated HTTPS push path that bypassed gitolite entirely.[3]

This was not primarily a failure to recognize malicious C. It was a failure to remember every route that could make C canonical.

The diff was loud; the authority looked familiar

The first commit was almost theatrically conspicuous. It touched the zlib output-compression path, introduced an HTTP_USER_AGENTT trigger with an extra “t,” searched for a vendor name associated with exploit acquisition, and called PHP’s own evaluator on attacker-controlled text. The full change fit in 11 added lines. Its message said “typo”; its behavior said remote command channel.[2]

Putting Lerdorf’s name on the commit supplied a thin layer of social camouflage. But a Git author name and email address are metadata supplied by whoever creates the commit. They are not, by themselves, proof that the named person authenticated to the receiving server. Popov’s first reaction was nevertheless understandable: revert the change and revoke Lerdorf’s commit access. When the backdoor returned under Popov’s own name, that account-level explanation stopped fitting.[3]

The distinction matters in any source-control investigation. “Who does this commit claim authored it?” and “Which credential and protocol caused the server to accept it?” are different questions. A signed commit can connect content to a signing key; a hosting platform can connect a push to an authenticated principal; server and identity-provider logs can connect that principal to an event. A plain author field connects only text to text.

PHP’s public GitHub mirror preserved both malicious objects, including commit IDs c730aa26 and 2b0f239b, after their effects had been reverted.[1][2] That visibility helped reviewers inspect exactly what had happened. It did not explain how the writes passed authorization.

The missing log entry pointed at the wrong machine

The expected write route used SSH, gitolite, and public-key authentication. Popov checked gitolite and found entries for neighboring legitimate commits but no git-receive-pack records for the two malicious ones. The team reasonably treated that absence as evidence that git.php.net itself might have been compromised. On March 28, it announced that the self-hosted server would be discontinued and the existing GitHub mirrors made canonical; membership in the PHP GitHub organization would require two-factor authentication.[1][3]

The inference was useful because it triggered containment. It was not the final cause.

During the deeper review, Popov discovered that git.php.net intentionally exposed another write route. Contributors could push over HTTPS through git-http-backend, with Apache HTTP Digest authentication checked against the user database on master.php.net. That traffic never traversed gitolite, so its absence from gitolite’s logs was expected. Web access logs showed failed guesses followed by successful git-receive-pack requests for the rasmus and nikic usernames over this HTTPS path.[3]

The blind spot was architectural and organizational at once. One maintainer investigating the incident did not know the alternate push method existed. It used a different protocol, a different authentication mechanism, a different credential store, and a different audit trail, yet it could mutate the same authoritative repository. The visible control plane said “SSH keys and gitolite.” The actual control plane also included old web authentication coupled to a separate account system.

An independent follow-up by The Register captured the revised assessment: the team no longer believed the Git server itself had been breached and instead considered a leak of the master.php.net user database possible.[6] “Possible” is the important word. The logs proved successful password-authenticated HTTPS pushes. They did not prove how the attacker obtained the necessary credential material. Popov noted that the attacker guessed several usernames before authenticating successfully, which did not fit a simple, fully understood database-theft story.[3]

Legacy compatibility kept a weak credential format alive

The account system widened the concern. According to Popov’s update, master.php.net ran very old code on an old operating system and PHP version. Passwords were stored in a form compatible with HTTP Digest authentication—essentially a plain MD5 hash—because the Git and Subversion services still depended on that scheme. The postmortem did not establish a specific vulnerability in the account application, and it should not be rewritten into one. It did establish that an obscure compatibility requirement prevented the project from using a modern password hash.[3]

This is a common infrastructure trap: the weakest surviving consumer dictates the security properties of a shared identity store. An old service may look peripheral because few people use it. If it can still write source, publish artifacts, reset credentials, or mint tokens, it is not peripheral. Its compatibility needs are production security requirements, whether anyone has documented them that way or not.

PHP removed the dependency rather than trying to make the old arrangement respectable. It made both git.php.net and svn.php.net read-only, migrated the remaining active Subversion repositories to GitHub, reset all php.net passwords, moved the account application to a new PHP 8 system called main.php.net, adopted bcrypt password storage, and moved database access toward parameterized queries. Retiring HTTP-authenticated writes made the insecure Digest-compatible password representation unnecessary.[3]

The project also accepted a workflow change. GitHub went from mirror to canonical host, its organization membership supplied the write-access roster, and mandatory 2FA replaced part of the home-grown “karma” permission system.[1] This was not proof that hosted infrastructure is inherently safe. It was a decision to stop operating a little-used, poorly inventoried security-sensitive stack when the community was already doing much of its work elsewhere.

Why no poisoned PHP release appeared

The attacker reached canonical development history, which is serious. The attacker did not reach users. Two containment controls made that difference, while the public mirror strengthened the forensic record.

First, people were watching the stream of changes closely enough to catch a tiny but absurd diff and revert it quickly; that immediate reversion is why the code did not reach users. Second, the project treated uncertainty about repository integrity as a release problem: on March 30, it put releases on hold for two weeks while checking the root cause and scope.[4][5] The public GitHub mirror preserved the malicious objects and reversals for inspection, but visibility alone was not the stop. A source branch was not automatically a shipped interpreter.

The incident therefore argues against two comforting simplifications. Open source did not save PHP merely because the code was visible; visible code helps only when people or automation actually examine changes. Nor did repository compromise automatically mean every PHP installation was compromised; a release process with time, review, and an emergency stop separated a poisoned commit from a distributed artifact.

Those controls also explain why “the backdoor was obvious” is not a satisfying dismissal. A subtler change, a quieter maintainer identity, or a push timed around legitimate refactoring could have survived longer. Detection quality reduced this attacker’s dwell time. Removing the untracked write path reduced the next attacker’s opportunity.

The engineering review this incident still demands

The practical exercise is to draw every route that can change what users trust. Start with direct Git pushes and pull-request merges, then add bot accounts, CI tokens, release automation, package registries, container tags, documentation deployers, mirrors that can become primary, account-recovery systems, and any legacy protocol retained for one extension or one contributor. For each route, ask four questions:

What authenticates the write?
What policy approves it?
Where is the event logged and alerted?
What separate gate prevents it from becoming a release?

If two routes can mutate the same target, they should not silently offer different assurance. A legacy route that cannot support the current authentication, review, and telemetry standard needs a named exception, compensating controls, an owner, and a removal date. “Nobody uses it” is evidence for deletion, not for leaving it unmonitored.

Monitoring should also follow the protected object rather than one favored gateway. PHP’s investigators first searched gitolite because they believed gitolite mediated writes. A repository-level feed, immutable receive log, or out-of-band mirror comparison can reveal an accepted ref change regardless of whether it arrived through SSH, HTTPS, an automation token, or an administrator path. Gateway logs explain; canonical-state monitoring detects.

Finally, rehearse the stop. PHP could make the repository read-only, switch the canonical host, reset credentials, review history, and pause releases.[1][3][4] A project that cannot name who has authority to freeze pushes and publication during an incident has already made its recovery slower.

The enduring image from this episode should not be the misspelled header. It should be the empty log query: an investigator looked in the place where every accepted push was supposed to appear and found nothing. The correct response was not to trust the emptiness. It was to ask what other machine could still say yes.

cronfeed.work