A web archive is a recording, a package, and a replay engine

Internet Archive server racks at the organization's San Francisco headquarters in 2013. The hardware is not the software stack mapped here; it makes the final boundary visible: captured exchanges still need maintained physical storage. Photograph by Jason Scott, CC BY 2.0.[8]

Open a preserved webpage ten years after it was captured and the masthead may appear perfectly while the article beneath it remains blank. The HTML arrived. A script that was supposed to fetch the story did not. A screenshot would have retained the visible words but lost the links, menus and search. A folder produced by “Save Page As” might contain a plausible shell with no account of which remote responses built it.

Web archiving is difficult because a modern page is less like a document than a short performance. The browser requests an HTML shell, executes code, negotiates consent, calls APIs, selects images for its viewport, reacts to scrolling and may reveal more only after a person clicks. Preserving that performance requires several open-source layers with different responsibilities: Browsertrix Crawler or ArchiveWeb.page exercises the live site; WARC records network exchanges; WACZ packages those records with indexes and page metadata; ReplayWeb.page turns the package back into a browsable experience.

The stack works when those layers agree, but agreement is not the same as truth. A flawless replay can faithfully reproduce an incomplete crawl. A valid digest can prove that a package has not changed without proving that the live server ever sent the claimed bytes. An enormous WARC can contain thousands of responses without the one API call that made the page meaningful. The useful way to map this ecosystem is therefore by the evidence each layer owns—and the questions it cannot answer.

Image context: Jason Scott's 2013 photograph shows Internet Archive storage racks in San Francisco. Those machines are not evidence that any particular page was captured well. They represent a separate durability layer that begins after selection, capture, packaging and replay have done their work.[8]

Five contracts, not one save button

Stage	Open component or format	Contract it can establish	Failure it cannot settle alone
Scope and interaction	Browsertrix Crawler / ArchiveWeb.page	Which URLs and browser actions were attempted	Whether an unvisited state contained important material
Capture record	WARC 1.1	Which requests, responses and related metadata were stored	Whether the stored set is a complete representation of the page
Portable collection	WACZ 1.2	Where records, indexes, page entries and fixity data live in one package	Whether the captured content was authentic or editorially sufficient
Access	ReplayWeb.page	How archived requests can be resolved without a central replay server	Whether missing behavior was absent at capture or broken during replay
Comparison	Browsertrix QA	Where replay differs from what the capture browser observed	Whether the original crawl exercised the right states in the first place

This division matters operationally. A preservation team can replace a crawler while retaining WARC as the evidence boundary, or replace a viewer while retaining WACZ as the delivery boundary. It can also diagnose a blank page without calling the whole archive “broken”: the missing object may never have been requested, may be absent from the package index, or may be present but mishandled during replay.

Capture is an editorial act performed by a browser

Browsertrix Crawler runs a real Brave browser under Puppeteer control and records traffic through the Chrome DevTools Protocol. It can launch multiple browser windows, accept YAML configuration, start from seed lists, apply per-seed scope rules, block selected URLs, reuse browser profiles and run behaviors such as autoscroll or site-specific interaction. A crawl can be watched while it runs, which is useful because an apparently busy crawler may be trapped in a calendar, faceted search or other space that generates effectively endless URLs.[1]

Those capabilities do not remove curatorial judgment; they turn it into configuration. A domain-wide scope may follow too much. A prefix scope may miss a JavaScript route whose API lives on another host. Autoscroll may wake an infinite feed but still never open the “load more” control. Four parallel browser windows can shorten a job while changing rate-limit pressure or the sequence in which a site emits personalized content. The configuration is therefore part of the archival record, not disposable launch syntax.

ArchiveWeb.page offers the complementary capture lane. Its browser extension and Electron app let an operator record interactively while navigating a site. The application uses the Chrome debugging protocol, stores captures locally in IndexedDB and can export portable web archives.[2] This is often the clearer fit for a bounded experience: an exhibition with several nonlinear paths, an authenticated page the collector is authorized to preserve, or a state that requires a particular gesture. The human can see whether the carousel advanced and whether the map labels appeared.

The tradeoff is coverage. Interactive capture preserves the path the operator actually took; it does not silently discover every path not taken. Automated crawling offers repeatability and breadth; it does not know which state matters culturally or evidentially. A mature collection may use both: scheduled Browsertrix runs for the stable perimeter, then targeted ArchiveWeb.page sessions for interactions that require eyes and intent.

Credentials sharpen the boundary. A saved browser profile can make an authenticated page capturable, but cookies, headers, browser storage and personalized responses may contain secrets or personal data. Dedicated capture accounts, least privilege, explicit authorization, restricted raw-archive access and a review step before publication are not optional polish. The crawler can record what a browser sees; it cannot decide what an institution is entitled to retain or disclose.

WARC keeps the exchanges, not a finished page

The WARC 1.1 format is a sequence of typed records. A request record can preserve the HTTP request, a response record the returned message, a metadata record contextual information, and a revisit record a duplicate-detection event that refers back to previously stored payload. Required fields such as WARC-Date, WARC-Type and WARC-Record-ID identify the event and record; target URIs and concurrent-record links connect related evidence. Block and payload digests can make later byte-level change detectable.[3]

That model is deliberately lower-level than “the page.” Suppose an HTML response loads app.js; the script asks an API for story.json; the returned story refers to a responsive image; and a click asks for comments. Each transaction can become a separate WARC record. If the capture browser never clicks the comments control, there may be no comments request to record. If an API returns a transient error, WARC can faithfully preserve the error. If the same image was already stored, a revisit record can avoid writing the full payload again while retaining evidence of the later observation.[3]

This is the format's strength. It preserves provenance-bearing protocol material rather than flattening everything into a rendered bitmap. It is also why a WARC file size is a poor quality metric. More bytes can mean deeper coverage, duplicated noise, oversized media or a crawler trap. The meaningful questions are whether the required states were exercised, whether their dependencies were captured and whether another browser can resolve those dependencies from the record set.

Modern JavaScript keeps that last question open. Independent systems research presented at OSDI 2022 found that archives can retain large amounts of JavaScript while still replaying pages incorrectly, whereas end-state screenshots lose post-load interaction. The authors' experimental Jawa design reduced storage by 41 percent against the comparison technique on a one-million-page corpus while improving replay fidelity, but it did so by addressing nondeterministic execution rather than by declaring capture solved.[7] The broader lesson applies to today's tooling: stored code is necessary evidence, not a guarantee that future execution will follow the same path.

WACZ turns sequential evidence into a portable collection

A WARC is a durable record container, but a remote viewer should not have to scan gigabytes from the beginning to locate one response. WACZ adds a delivery contract. Version 1.2 defines a ZIP package containing WARC data under archive/, CDXJ lookup indexes under indexes/, a required pages/pages.jsonl navigation list and a root datapackage.json manifest. Each page entry carries at least a URL and timestamp; it may also carry a title, extracted text and aggregate size.[4]

The CDXJ index maps archived URLs and capture times to byte locations in the WARC data. Combined with independently compressed WARC records, that makes range-based access practical: a viewer can request the relevant slice instead of transferring the complete collection. The manifest inventories resources with paths, sizes and hashes. An optional datapackage-digest.json hashes the manifest, extending fixity checking across the package.[4]

WACZ therefore joins objects that serve different readers. WARC remains the protocol evidence. CDXJ serves lookup. pages.jsonl supplies human navigation and search context. The manifest supplies inventory and fixity. Zipping them together makes a collection easier to move, publish and open without redefining the underlying capture format.

Two boundaries should stay visible. First, fixity is not authenticity: a matching hash shows that the package still matches its manifest, not that the origin server vouched for the content or that the collector chose an adequate scope. Second, portability is not preservation. A WACZ still needs redundant storage, access control, routine fixity checks, documented retention and tested restore paths. The server racks in the cover photograph belong to that later layer. No packaging specification replaces maintained copies.

Replay is controlled request resolution

ReplayWeb.page is an open-source viewer that runs client-side in a modern browser. It can load static web archive files from local or remote storage, search page and URL metadata, use included extracted text for full-text search and incrementally load large files. Because the viewer does not require a central replay server, a collection can be hosted as static data and embedded in another site.[5]

The important behavior is what it does not do: a sound replay path should satisfy an archived page's requests from captured records rather than quietly fetch missing material from today's live web. Live leakage can make a broken archive look healthy, mix two historical moments and disclose a reader's request to a third party. Testing with outbound network access blocked is one of the simplest ways to expose that illusion.

Replay also has its own failure surface. URL rewriting can fail. A request may be present under a timestamp the viewer does not select. A security rule in the modern browser may prevent behavior that worked at capture time. A package served remotely may not support the access pattern the viewer needs. When a page fails, the operator should ask in order: Was the interaction performed? Was the resulting response written to WARC? Is it indexed in WACZ? Can the viewer resolve it? That sequence is much more useful than recrawling blindly.

QA compares two observations, then a person judges them

Browsertrix can run a QA crawl against an existing WACZ and compare replay with what the browser observed during capture. With --screenshot view and --text to-warc, the analysis can compare viewport images and extracted text; page information also exposes resource counts and response status groups. The resulting scores help surface pages whose replay diverges sharply from capture.[6]

These are triage signals, not acceptance proofs. A high screenshot similarity can hide a menu that never opens. Matching text can coexist with broken chronology or incorrect media. Equal resource counts do not prove that the resources are the right versions. Conversely, animation, rotating banners or delayed fonts can depress a visual score even when the preserved content is usable. Automated comparison narrows the review queue; a collector still decides what “good enough” means for the collection's purpose.

A defensible acceptance set is small enough to inspect and varied enough to break assumptions. Include a static article, a client-rendered route, an infinite or paginated view, a page with third-party assets, and—only when authorized—an authenticated state. For each representative page:

Record seeds, scope rules, exclusions, behaviors, browser profile policy and the exact crawler release or container digest.
Capture once, then replay with live-network egress blocked.
Check text, images, internal navigation, timestamps and the one interaction that gives the page its meaning.
Inspect missing requests in the WARC and WACZ index before changing crawler behavior.
Store the accepted WACZ with its configuration, review notes, fixity record and restore-tested copies.

The team boundary follows the collection. A researcher preserving a dozen hand-selected experiences may be well served by ArchiveWeb.page and careful session notes. A small library repeating several scoped crawls needs Browsertrix configuration under version control, storage monitoring and a review queue. A large program also needs scheduling, access roles, sensitive-data handling, retention policy and operators who can distinguish crawler failure from replay failure. Adding browser windows is not a substitute for that operational maturity.

The open ecosystem's real advantage is not a magic capture button. It is inspectable separation. Browsertrix and ArchiveWeb.page expose what the collector asked a browser to do. WARC exposes what crossed the network. WACZ exposes how the evidence was indexed and packaged. ReplayWeb.page exposes whether those records can support a visit now. QA exposes disagreement between two observations.

A trustworthy web archive keeps those claims separate long enough to test them. It can say what was attempted, what was recorded, what survived unchanged and what still replays—and it can admit when those are four different answers.

cronfeed.work