DeepSeek opened 3FS, but the fast path still begins in the machine room

Three all-flash nodes in a photographed media-storage cluster. This is not a 3FS deployment; it makes visible the dense NVMe hardware behind the same class of disaggregated-storage problem.[7]

The glamorous unit of AI infrastructure is the accelerator. The unit that quietly wastes it can be a file: a training sample that arrives late, a checkpoint that takes too long to land, or a recovery job that competes with the next run for bandwidth.

DeepSeek’s Fire-Flyer File System, better known as 3FS, moves that problem into the foreground. Its public design treats datasets, intermediate outputs, checkpoints, and even inference KV cache as traffic on one shared storage plane built from NVMe SSDs and an RDMA network. Applications see files rather than storage locations; the client derives chunk placement from filesystem metadata, then contacts the relevant storage services across the fabric.[1]

The repository became public on February 27, 2025. As of July 26, 2026, its main branch includes maintenance through May 7, when the latest commit fixed monotonic versioning for a chain table.[1] That continued work matters more than the launch-week novelty. It shows 3FS as living systems software, where a small ordering invariant can be as consequential as a spectacular throughput graph.

It also clarifies the limit of the open-source story. DeepSeek has opened the design and implementation. It has not turned 3FS into a storage appliance that any model team can install over a weekend. The published fast path begins with racks of flash, a carefully operated RDMA fabric, separate metadata and monitoring services, and application work at the client boundary. The export is a serious infrastructure recipe—not instant storage independence.

A filesystem arranged around expensive waiting

3FS grew inside High-Flyer’s Fire-Flyer II AI cluster, where storage and compute were separated but joined by a high-bandwidth network. High-Flyer’s current site still presents the filesystem alongside its training scheduler, optimized operators, and collective-communication layer, while its system paper describes a 10,000-A100 cluster designed through hardware–software co-design.[2][3] That ancestry is important: 3FS was not conceived as a general office file share. It was built to keep a large accelerator fleet from waiting.

The public system has four principal parts. A cluster manager maintains membership and data-placement state. Stateless metadata services implement file operations while keeping metadata in FoundationDB. Storage services manage chunks on local SSDs. FUSE and native clients present the files to applications. The services communicate across InfiniBand or RoCE rather than an ordinary best-effort application network.[1]

This separation creates two useful properties. First, clients can send metadata requests to any metadata service because the durable inode and directory-entry state lives in the transactional key-value store. FoundationDB’s default transaction model detects conflicts and gives the metadata layer a strictly serializable basis for operations such as create, link, unlink, and rename.[1][5] A Chinese source-level analysis from Alibaba Cloud’s developer community highlights the same choice: 3FS pushes cross-shard filesystem transactions into FoundationDB, then keeps the metadata service itself horizontally replaceable.[4]

Second, file data is divided into equal-sized chunks, striped across replication chains, and stored on multiple SSDs. 3FS uses Chain Replication with Apportioned Queries, or CRAQ. Writes move through every target in a chain; reads can be apportioned among replicas once their version state is safe to serve. The original CRAQ work was designed for read-mostly workloads because adding replicas can expand read capacity without giving up strong consistency.[1][6] That shape fits model training unusually well: enormous shared datasets are read repeatedly, while checkpoints and transformed datasets still need durable, understandable file semantics.

The point is not that files are newly fashionable. It is that 3FS makes the file interface sit on top of an explicitly distributed data plane. A CSV or Parquet loader can see a familiar path while the storage layer chooses chains, spreads chunks, balances reads, and recovers replicas underneath.

The 6.6 TiB/s number has a machine room attached

The headline result in the repository is approximately 6.6 TiB/s of aggregate read throughput. The boundary around that number is unusually revealing. DeepSeek reports a stress test using 180 storage nodes, each with sixteen 14-TiB NVMe SSDs and two 200-Gbps InfiniBand interfaces. More than 500 client nodes, each with a 200-Gbps interface, drove the reads while training traffic remained on the system.[1]

This is compelling evidence that the architecture can aggregate flash and network bandwidth at fleet scale. It is not a universal filesystem speed. The published figure is a large-cluster aggregate read test; it does not by itself report end-to-end model throughput, checkpoint completion time, tail latency during a failure, cost per usable terabyte, or performance on a smaller Ethernet installation. The hardware and traffic pattern are part of the result, not footnotes to remove from it.

The repository offers two other windows, and each uses a different setup. A GraySort run used 25 storage nodes and 50 compute nodes to sort 110.5 TiB into 8,192 partitions in 30 minutes 14 seconds, an average of 3.66 TiB per minute. Its KV-cache example reports peak read throughput of up to 40 GiB/s across clients whose nodes had one 400-Gbps interface, but the README does not provide enough workload detail to translate that observation into a general inference-capacity claim.[1]

Read together, the tests show breadth: bulk reads, data processing, and an inference-oriented cache path. They do not form one comparable benchmark suite. The clusters, operations, client counts, and denominators change. Any buyer or operator treating “6.6 TiB/s” as a procurement specification would first need to reproduce the relevant access pattern on its own drives, NICs, topology, replication factor, and failure policy.

The historical boundary matters too. The Fire-Flyer II paper reports the surrounding system on PCIe A100 GPUs and an integrated compute–storage network.[2] 3FS is therefore evidence that a Chinese lab developed valuable infrastructure software under compute-cost pressure. It is not evidence that the original stack was independent of Nvidia hardware. Its portability claim lives at the storage interface; the published peak lives in a particular facility.

The familiar mount and the fast client are different products

3FS offers a FUSE client because a mounted filesystem is a powerful migration tool. Existing applications can continue to open paths, inspect directories, and read file-based datasets. That is the friendly lane.

DeepSeek’s own design notes also document where that lane ends. In the team’s benchmark, FUSE saturated at roughly 400,000 4-KiB reads per second because requests passed through a shared multithreaded queue protected by a spin lock. Linux 5.x FUSE could not issue concurrent writes to one file, and small, unaligned random reads from training samples failed to use all available SSD and network bandwidth.[1] Those are not minor cases for AI workloads; they are exactly the patterns a data loader can create.

The performance lane is a native interface called USRBIO. Applications still use the ordinary file path for metadata operations, then register the file descriptor and submit asynchronous I/O through shared-memory rings. Large registered memory regions let the client move data without another copy, while batching reduces the cost of many small requests. The Alibaba analysis likewise identifies a zero-copy RDMA path that bypasses the ordinary FUSE I/O route for performance-critical access.[1][4]

That split resolves an apparent contradiction in the project’s pitch. There is no new storage namespace to learn, but peak performance can still require application integration. Teams can adopt the mount first and move selected loaders or checkpoint writers onto USRBIO later. They should not benchmark the easy path and assume it represents the fast one—or integrate the native client and assume it preserves every POSIX behavior.

The design notes state one such semantic boundary plainly: 3FS does not track read-only file descriptors in the same way a local filesystem would, because training jobs can open enormous numbers of files and do not depend on deferred deletion for those readers. File length can also be only eventually consistent during concurrent writes until close or fsync forces the metadata service to query the final chunk.[1] These are deliberate system choices, not defects, but they make workload qualification essential.

Open source does not remove the deployment tax

The manual setup guide is the best antidote to a frictionless reading of 3FS. Its example cluster has one metadata node and five storage nodes. Each storage node is specified with 512 GB of memory, sixteen 14-TB SSDs, and RoCE; that is 80 drives and roughly 1.12 PB of raw decimal capacity before replication. The guide recommends dedicated production nodes for FoundationDB and ClickHouse, asks operators to verify RDMA with ib_write_bw, formats every NVMe device as XFS, raises the asynchronous-I/O limit, creates multiple storage targets per disk, and generates a three-replica chain table with a placement solver.[1]

The current build also carries a compatibility warning born from std::shuffle: operators must choose a g++10 or g++11 shuffle method and keep that choice consistent across future builds for an existing cluster. Recent maintenance has fixed timeout handling, truncate/extend synchronization, and chain-table version ordering.[1] The pattern is familiar to storage engineers. Performance depends on invariants that cross source code, compiler behavior, network configuration, on-disk state, and rollout discipline.

This is the supply-chain significance of 3FS. China’s AI stack is often narrated vertically from accelerator to compiler to model. 3FS adds a lateral constraint: every accelerator must be fed, every failed run must leave recoverable state, and every checkpoint must cross a storage fabric without turning scarce compute into idle capital. A shared storage plane can amortize that work across model teams and, in principle, across heterogeneous compute fleets.

But the software does not commoditize the machine room by itself. It shifts value toward operators who can assemble flash density, RDMA, placement, metrics, transactional metadata, and workload-specific clients into one reliable service. For a hyperscale lab, that control can be strategic. For a smaller team, managed object storage or a mature commercial parallel filesystem may remain the better trade, even if its peak graph is less dramatic.

What would prove 3FS travels

The next proof is not a larger number from the original cluster. It is independent operation under different constraints.

First, adopters need reproducible results at modest scale: dataset-random-read, parallel-checkpoint, recovery, and metadata tests on clusters far smaller than 180 nodes, with p95 and p99 latency as well as throughput. Second, the project needs upgrade and rollback evidence across compiler settings, FoundationDB versions, chain-table changes, and mixed client revisions. Third, the KV-cache path needs an explicit workload definition and comparisons against DRAM, local NVMe, and purpose-built disaggregated-cache systems on cost per accepted token—not only bytes per second.

The falsifier is straightforward. If organizations can clone the code but still require private patches, the original network topology, and unusually deep in-house storage expertise to reach stable performance, 3FS will remain an influential design reference rather than a broad supply-chain layer. If its public tests, packaging, and operational playbooks become portable across hardware vendors and cluster sizes, it can become something larger: a reusable answer to the unglamorous question underneath every AI run—how to keep expensive compute from waiting on a file.

cronfeed.work