Webmin replaced its build server—and restored the backdoor

RobH photographed Wikimedia's Tampa server cluster in January 2009. These are not Webmin's machines; the archival photograph makes the incident's hidden recovery problem tangible: replacing hardware or an operating system does not make the workspaces restored onto it trustworthy. RobH / Wikimedia Commons, CC BY 2.5; resized from the original.[7]

On September 10, 2018, Webmin replaced a development build server later determined to have been compromised with a newly installed CentOS 7 machine. The new operating system was clean. The build directory copied from the old server's backups was not. Inside it, a modified password_change.cgi still contained a backdoor, and later official packages carried that file forward.[1]

The episode became public in August 2019 as a root-level remote-command vulnerability in Webmin, the web interface used to administer Unix-like systems. Its most durable lesson is not simply that build servers need protection. It is that reinstalling a machine and restoring its mutable workspace are opposing actions. One removes untrusted state; the other may reintroduce it.

That distinction was unusually consequential here because three objects that users often collapse into “the release” disagreed:

Webmin's checked-in GitHub source did not contain the malicious code.
The local directory used to assemble releases did.
Packages offered through the project's official SourceForge download path inherited the local directory's state.[1][2][5]

The photograph above shows the back of Wikimedia's servers in Tampa in 2009, not Webmin's build host.[7] Its cables, ports, fans, power supplies, and chassis supply useful physical scale. A build server is not a neutral arrow between source and download. It is a stateful computer, and a backup can preserve its unreviewed state as faithfully as its intended inputs.

One script was modified twice

Webmin's reconstruction begins in April 2018, when an unknown attacker exploited the development build server and altered password_change.cgi. The change placed attacker-controlled input inside Perl's qx// execution construct, which runs its contents through a shell. In Webmin 1.890, the resulting path could be reached without authentication under the default configuration and execute commands with root authority.[1][5]

The project's account says the attacker also set the file's timestamp back, preventing the change from appearing in the diffs used in that workflow. The file was later restored from the checked-in GitHub version. Then, in July 2018, it was modified again. This second form entered Webmin 1.900 and subsequent packages through 1.920, but it sat behind a configuration condition: changing expired passwords had to be enabled for the vulnerable path to be reachable.[1]

That version distinction matters. “Installed Webmin” was not a single exposure state:

1.890 from the affected official package path: remotely exploitable in a default installation.
1.900 through 1.920 from that path: vulnerable when the expired-password-change feature was enabled.
A build made from the corresponding clean GitHub source: not known to contain the injected build-directory code.[1][3][5]

The CVE Program record for CVE-2019-15107 describes command injection through the old parameter of password_change.cgi. The Webmin security page still displays CVE-2019-15231 beside its incident account, but the CVE Program now marks that second identifier rejected as a duplicate and directs readers to CVE-2019-15107.[1][3][4] The bookkeeping changed after disclosure; the operational distinction between the default-exploitable 1.890 package and later configuration-dependent packages did not.

A clean repository did not imply a clean release

The public repository was the wrong place to reproduce the bug. Webmin developer Joe Cooper told The Register that the exploitable code had never been in the project's GitHub repositories. Looking at those files could therefore reassure an investigator while the packages users actually downloaded remained malicious.[5]

This mismatch was not as simple as “GitHub archive good, SourceForge archive bad.” In a May 2019 mailing-list exchange—three months before disclosure—a user asked why the GitHub and SourceForge archives differed so much. Project author Jamie Cameron answered that the SourceForge package was the official release and that GitHub's automatically obtained tree had not been packaged properly.[2] Legitimate release work created expected differences: generated files, selected modules, metadata, or other packaging output could make artifact equality an invalid test.

That left a dangerous authority gap. The repository defined reviewed source, but it did not completely define the shipped artifact. A long-lived local directory supplied the missing transformation. Because that directory was both mutable and authoritative, an attacker could alter the release without creating a malicious commit for reviewers to see.

SourceForge was the distribution venue, not necessarily the intrusion point. The maintainer account places the compromise on Webmin's build infrastructure, after which affected packages were published through the normal official channel.[1][5] This is precisely why a familiar HTTPS download URL is insufficient provenance: it authenticates the site serving the bytes, not the history of how those bytes were assembled.

The incident separates four assurances that should be tested independently:

Source integrity: does the reviewed commit contain the intended code?
Build-input integrity: did the job start only from declared source, dependencies, and tools?
Artifact integrity: is the package the output of that job rather than residue from a mutable workspace?
Release authority: did an approved identity promote and sign that exact artifact?

Webmin's Git history offered evidence for the first assurance. The local packaging process broke the chain before the second and third could inherit it.

The clean-machine migration preserved the wrong state

On September 10, 2018, the old build server was decommissioned and replaced. Copying the existing build directory from backups made operational sense: it preserved the machinery needed to keep shipping. It also treated the directory as business data rather than potential incident evidence. The malicious file crossed into the new environment without the attacker needing to compromise that environment again.[1]

This is the postmortem's sharpest mechanism. A clean install changes the trust status of the operating-system layer. It says nothing about restored application state. Build workspaces, caches, package staging directories, compiler wrappers, CI plugins, and signing material all sit above that layer. If their provenance is unknown, “restore” is not the final step of recovery; it is a new input requiring validation.

The move also narrowed later forensics. Cameron told The Register that the original machine had been decommissioned, eliminating logs that might have shown how the attacker entered. The project could reconstruct when the file appeared, where it differed, and how it propagated, but not the initial access path.[5] Any stronger attacker narrative would go beyond the available record.

A safer recovery would have treated the old workspace in two incompatible ways at once—even if, at the time of migration, it was only material that later proved to be incident evidence:

Preserve it as evidence: snapshot it read-only, record hashes, and retain logs for comparison.
Reject it as a build input: create a new empty workspace from an authenticated commit and a reviewed build definition.

The same bytes should not move directly from the evidence lane into the production lane. Backups optimize for continuity; they do not certify that the saved state predates compromise.

Disclosure exposed two different blast radii

A public exploit surfaced at DEF CON on August 10, 2019. Webmin's maintainers said they learned of it on August 17, removed the code, and released Webmin 1.930.[1][5] Ars Technica reported more than 900,000 downloads across the affected releases. That is a distribution count, not proof that every download became an exposed installation or that every exposed host was exploited.[6]

The response therefore needed two inventories.

The first was an artifact inventory: which version was obtained, from which channel, and with which configuration? Upgrading to 1.930 removed the known vulnerable code for systems not already compromised. The second was a host-compromise inventory: had the remote-command path been exercised on an Internet-reachable management service with root authority? For a host where exploitation could not be excluded, replacing one Webmin package could not revoke commands already run, persistence already installed, or credentials already read. Independent contemporary analysis accordingly emphasized full host rebuild and credential rotation as the conservative recovery response.[6]

Webmin's own project response addressed the release process. It changed the build to use only checked-in GitHub code instead of a locally synchronized directory, rotated passwords and keys accessible from the old build system, and audited the preceding year's GitHub commits for other suspicious changes.[1] Each action closed a different route: undeclared build state, possible credential exposure, and unreviewed or malicious source changes.

Make deletion part of the release design

For a small project, the answer is not an elaborate supply-chain platform that nobody can operate. It is a release path whose important state can be deleted before every build.

A defensible minimum looks like this:

Resolve an approved tag to an exact commit and record that commit ID.
Create a new workspace; do not reuse or restore the previous job's working tree.
Fetch declared dependencies and toolchains from pinned or integrity-checked references.
Build the package and record a manifest of the inputs and output digest.
Run a second comparison—rebuild when practical, or at least unpack the artifact and reconcile every expected difference from the source tree.
Let a separately controlled release identity sign or promote the resulting digest, not an arbitrary path on the worker.
Retain the manifest, logs, signature, and artifact outside the worker's administrative domain.

Reproducibility strengthens this chain but does not replace source review. Two builders will faithfully reproduce a malicious commit if that commit is the declared input. Conversely, a byte-for-byte comparison may need normalization when archives contain timestamps or generated metadata. The useful control is not a ritual command; it is an explained correspondence between reviewed source and published bytes.

There is also a practical limit. A two-maintainer project may not have independent release engineers, hardware signing devices, or two isolated builders. It can still make its worker ephemeral, keep the publishing token out of pull-request jobs, require a reviewed tag, inventory packaged files, and save a signed checksum somewhere the package host cannot silently rewrite. Those modest controls directly address the Webmin mechanism.

The remediation test is concrete: delete the release worker, clone the approved commit into an empty environment, and execute the declared build. If that produces the same explainable artifact while a separate identity controls publication, copying an old workspace from backup can no longer preserve this particular attack path.

The known propagation path required no second clever intrusion into Webmin's new server. It inherited the earlier intrusion's result through an ordinary recovery action. That is what makes the incident valuable: the dangerous step looked like continuity. A trustworthy release begins by deciding which state deserves to survive.

cronfeed.work