A short Spack spec can unfold into an exact build graph

The Spack development team at Lawrence Livermore National Laboratory in 2023: Greg Lee, Matt Legendre, Todd Gamblin, Peter Scheibel, Greg Becker, and Tammy Dahlgren. Photograph by James Chalabi.[8]

hdf5 +mpi %gcc target=x86_64_v3 looks like an installation command. Architecturally, it is closer to a question.

The line asks for HDF5 with its MPI variant enabled, built with GCC for a particular processor target. It does not name the HDF5 release, the GCC release, the MPI implementation, every transitive dependency, or the versions and variants of those dependencies. It also says nothing about which libraries a computing center has already installed, which provider that center prefers, or whether a compatible binary exists in a cache.

Spack's central job is to turn that partial request into one exact answer. It does so in stages: parse an abstract spec, combine it with package metadata and site policy, solve a dependency graph, write the selected graph to a lockfile, and either build or retrieve artifacts identified by the resulting graph. The useful thing about this architecture is not that it abolishes build complexity. It assigns different kinds of complexity to different records.

The input is a constraint set

Spack calls its package-description language a spec syntax. A user can constrain a package name and version, enable or disable variants, choose a compiler, state a target, and add constraints on dependencies. The documentation's fuller examples are compact but expressive: @ introduces a version, + and ~ switch Boolean variants, % selects a compiler, and ^ adds a dependency constraint.[1]

That syntax distinguishes an abstract spec from a concrete one. An abstract spec may leave many choices open. A concrete spec has one version, one compiler configuration, one architecture, and a fully resolved dependency DAG—directed acyclic graph. The same short request can therefore have several valid concrete answers.[1]

Virtual dependencies make the distinction especially visible. A package can require an MPI interface without requiring one implementation in every installation. Open MPI, MPICH, or a site-provided MPI may satisfy that role, subject to the rest of the graph and local configuration. The token +mpi expresses a capability requirement; it does not by itself settle who provides the capability.

This is the first boundary to keep clear in reviews. A command copied from a shell history shows what someone requested. It does not necessarily show what Spack selected.

Package recipes define what is possible

Each Spack package recipe is a Python module. Its declarative portion lists facts such as available versions, variants, conflicts, and dependencies; its methods contain the imperative instructions needed to configure and build the software. During a build, the selected concrete spec is available to the package so that its logic can respond to the chosen compiler, variants, and dependencies.[2]

That separation has a practical consequence. A recipe should capture knowledge about the software: for example, that enabling one feature adds a dependency, or that a release is incompatible with a compiler range. It should not have to encode every institution's module layout or preferred MPI.

Site choices live in configuration. In packages.yaml, administrators can register external installations, choose providers for virtual dependencies, prefer particular versions or targets, and mark packages as not buildable when only a supplied installation may be used.[4] concretizer.yaml adds solver-wide policy, including whether existing installations or build-cache entries may be reused and how processor targets are selected.[3]

The package recipe and site configuration thus create the search space together. The former describes legal builds in general; the latter narrows those possibilities for a machine, facility, or workflow.

Concretization is the hinge

Concretization is the moment when Spack replaces open choices with a single graph. The solver must reconcile the root request with dependency rules, conflicts, compiler availability, target compatibility, provider choices, external packages, preferences, and the constraints that dependencies impose on one another.

The scale of that operation is easy to underestimate because the input stays short. A request for one application can expand into dozens or hundreds of nodes, and a decision low in the graph can rule out choices above it. A 2024 paper about Los Alamos National Laboratory's then-current Spack-based production environments describes the Clingo-based concretizer as translating package constraints into declarative logic and solving the combined problem.[7]

Two settings show why concretization is policy rather than mere dependency lookup. With reuse enabled, Spack can consider compatible installed specs and build-cache contents while choosing an answer; with reuse disabled, it performs a fresh solve without allowing those existing concrete specs to determine the result. Separately, environment unification can require one concrete configuration per package across an environment or permit multiple configurations when roots need incompatible variants.[3][5]

Neither choice is universally correct. Reuse can save enormous build time, while a tightly controlled release process may value obtaining the same answer regardless of what happens to be available today. Unification can produce a more coherent environment, while some collections genuinely require two incompatible instances. The architecture exposes these as decisions instead of burying them in a build script.

The lockfile preserves the answer

A Spack environment has two records with different duties. spack.yaml is the manifest: it contains root specs and configuration—the request and the rules around it. spack.lock contains the fully configured, concretized specs selected from that input.[5]

This split is worth preserving in version control. A manifest alone is an invitation to solve again. If Spack, package recipes, external configuration, compiler discovery, or available providers change, a later solve may produce a different valid graph. A lockfile carries forward the previously selected graph. The environment documentation notes that creating from a lockfile on a compatible system initially guarantees the same concrete specs, whereas creating from a manifest may reconcretize differently.[5]

The qualification “on a compatible system” matters. A lockfile can record that a node was built for a target with a particular compiler and dependency graph. It cannot make an absent external library appear, repair a broken compiler installation, reproduce an unrecorded kernel assumption, or promise that the resulting program passes its tests. It freezes resolution, not the physical universe around the build.

That makes the two-file design useful in an incident. A diff in spack.yaml asks, “Which requirement or policy changed?” A diff in spack.lock asks, “Which concrete decisions changed as a result?” Mixing those questions into one hand-written installation script makes both harder to answer.

The graph becomes an identity

Spack needs to let incompatible builds coexist. HDF5 built with one MPI provider is not necessarily interchangeable with HDF5 built with another, even if both directories could be labeled “hdf5-1.x.” Compiler, architecture, variants, and dependencies are part of what the installation is.

Spack derives a DAG hash from the concrete spec. Binary-cache metadata uses that identity so builds with the same package name and version but different dependency graphs remain distinct. Current build-cache documentation describes manifests named with a concrete spec's DAG hash and content-addressed blobs protected by SHA-256 checksums.[6]

The hash does two jobs. In the installation store, it prevents distinct configurations from collapsing onto one human-readable name. In a build cache, it lets Spack ask for an artifact matching the resolved graph rather than for any archive labeled with the same upstream version.

This also separates resolution from materialization. The concretizer decides what the graph is. Installation decides how to obtain each node: reuse an installed spec, fetch a matching binary, or build from source. A cache miss is therefore not automatically a solver failure. The requested identity may be perfectly clear even when no artifact with that identity is available.

Production scale makes policy visible

The Los Alamos case study is valuable because it records where this model becomes operational work. The paper describes a spack.yaml of roughly 1,400 lines for a Cray programming environment, with about 85 percent devoted to pinning versions and variants. Its large environment could take as long as 30 minutes to solve. The team disabled reuse because changing cache contents could otherwise change the concrete answer, but then used the resulting binary cache in continuous integration, where installing the environment took about 20 minutes.[7]

Those numbers are not a current universal benchmark. They describe LANL's TCE2 production system and the Spack behavior and hardware reported in 2024. They are still revealing. The cost moved to a place the release team could inspect: a long manifest, an explicit solve, a lockfile, and a cache assembled from approved outputs. The same paper reports that Spack's package definitions and build logic generally served the project well and that binary packages were a particular strength.[7]

In other words, a declarative interface does not guarantee a small configuration. It makes the configuration stateable. At modest scale, that may feel like convenience. At production HPC scale, it becomes governance: someone must decide which variation is allowed, which result is promoted, and when a graph should be solved again.

Read failures at the layer that produced them

Spack is easier to debug when teams retain the architecture's stages.

An unexpected MPI provider points first to the abstract constraints and site provider policy.
An unsatisfiable solve points to conflicts among package rules, compilers, targets, externals, or environment unification.
A surprisingly large rebuild suggests that some concrete node changed and therefore changed identities downstream.
A cache miss says that the selected concrete artifact was not found; it does not by itself say the selection was wrong.
A program that installs but fails at runtime has crossed beyond resolution into build, ABI, environment, or application testing.

That vocabulary prevents one common failure mode: calling every problem “Spack chose the wrong package.” Often Spack chose exactly what the recorded constraints permitted. The harder question is whether those records expressed the organization's real policy.

Spack began at Lawrence Livermore National Laboratory in 2013 in response to the multiplying configurations of high-performance computing software. A decade later, LLNL photographed six members of its development team in a computing facility—the human maintenance context behind a tool that is often encountered only as terse terminal syntax.[8]

The short spec at the beginning of this article remains useful precisely because it is incomplete. It lets a scientist state what matters locally without hand-writing the entire build graph. But its brevity is made safe only by the layers behind it: package recipes delimit valid builds, site files express local reality, the concretizer selects one DAG, the lockfile records it, and the hash gives it an artifact identity. Spack does not make the dependency matrix disappear. It turns the matrix into an answer that can be reviewed, reproduced on compatible systems, cached, and—when necessary—disputed.

cronfeed.work