DeepServe’s serverless front door opens onto a stateful machine

Zhang Ping’an announces CloudMatrix384 at Huawei Cloud Ecosystem Conference 2025 in Wuhu, Anhui, on April 10, 2025. This company-issued event photograph supplies the physical launch context for the SuperPod named in DeepServe; it does not show a DeepServe experiment or machine room. Huawei Cloud.[5]

“Serverless” is a promise about what a customer does not have to manage. It is not a description of what disappears. Behind a chat-completions request, a serving platform still has to place model weights, retain or reconstruct a conversation’s key-value cache, coordinate multiple accelerators, detect stalled workers, and add capacity before a traffic spike turns into a queue. The interface may be stateless; the machine behind it emphatically is not.

That tension makes DeepServe: Serverless Large Language Model Serving at Scale worth watching. DeepServe is a Huawei Cloud system developed with Peking University and reported at the 2025 USENIX Annual Technical Conference. Presenter Junhao Hu says at the outset that the talk will emphasize the design judgments learned while putting the system into production, rather than treating it as a sequence of benchmark wins. The paper describes more than a year of operation on a large Huawei Ascend NPU cluster, serving fine-tuning, agent, and model-inference APIs.[1][2]

The 16-minute English-language recording was made and uploaded by USENIX. Watch for four linked ideas: a request–job–task abstraction, the modular FlowServe engine, routing between colocated and disaggregated prefill/decode workers, and an elasticity path built from already-warm software and already-nearby weights. Together they show what “serverless” has to conceal without pretending the concealed work is simple. The timestamps below are viewing landmarks, not a transcript.[1]

0:18–3:20 — the state arrives before the architecture diagram

Hu first places DeepServe inside a broader internal platform that combines inference, post-training, and agent serving on a shared resource pool. He then names the substrate: Huawei’s Ascend accelerators and the CloudMatrix384 SuperPod, whose 48 nodes contain 384 NPUs connected by a much faster fabric inside the pod than the scale-out network between pods.[1][2] This is more than a hardware preface. It establishes that a serving design is inseparable from the distances over which it moves tensors and weights.

Around 2:36, the talk turns that substrate into three problems. Models have become distributed systems rather than single-node programs. Reusing a key-value cache makes inference stateful. Demand changes, so the system must grow and shrink without making a new instance wait through a conventional cold start. Those problems pull in different directions: sending a request to the least-busy worker improves load balance, sending it to the worker holding the useful prefix preserves computation, and creating a fresh worker may strand both advantages.[1][2]

This is the first useful correction to the “serverless” label. The customer can submit an HTTP request without naming a machine, but the provider cannot schedule that request as if every machine were interchangeable. A prior prompt may already exist as cached tensors on one task executor. Model weights may be warm in host memory near another. A third executor may have the shortest queue. DeepServe’s job is to choose among those physical facts while keeping them outside the API contract.[2]

3:24–8:18 — request, job, and task divide user intent from scaling units

At about 3:24, Hu introduces the talk’s simplest and most durable abstraction. A request is the external trigger. One or more jobs interpret that request as chat, fine-tuning, or another service. Each job creates tasks, the smallest units of work; task executors are the smallest units the platform scales. A colocated chat engine may need one task, while prefill/decode disaggregation turns the same user request into separate prefill and decode tasks.[1][2]

The hierarchy is deliberately plain because, Hu explains, many of the system’s developers come from algorithm or data backgrounds rather than systems engineering. Plainness also creates a control boundary. Job executors can understand the workflow and dispatch work, while task executors can specialize and scale horizontally. A relational tensor cache spans those executors as a shared data plane for key-value state. The abstraction does not erase distribution; it gives distribution a vocabulary that the autoscaler and scheduler can act on.[1][2]

Two operational details late in this section keep the diagram honest. Around 7:39, Hu describes inference processes that do not crash but simply hang after accepting a request. Health checking therefore has to detect the absence of an output and deliberately kill the stuck component. Around 8:00, he notes that a large decode instance may require gang scheduling and even the eviction of other instances to assemble enough contiguous capacity.[1] In the paper’s fault model, failed job or task executors are rebooted and traffic is redirected to redundant peers; the relational cache holds append-only, recomputable “soft” state rather than paying for a complex consistency protocol.[2]

That is serverless from the caller’s side and state management from the operator’s side. The platform can hide worker identity only because it has explicit machinery for health, replacement, cache loss, and coordinated allocation.

8:22–13:05 — FlowServe keeps the NPU busy by moving everything else off its clock

The FlowServe section begins with three design principles: modularity inspired by microkernels, NPU-centric execution, and single-program-multiple-data parallelism. “Microkernel-inspired” here does not mean FlowServe is an operating-system kernel. It means tokenization, scheduling, tensor-cache management, networking, and execution are separated enough to evolve or scale without turning the entire serving engine into one indivisible process.[1][2]

The implementation choice at roughly 9:26 is revealing. The team kept much of the fast-changing control plane in Python, then moved performance-critical tensor transfer and cache swapping into C++. This is not a language beauty contest. It separates code that benefits from rapid iteration from code whose delays can leave expensive accelerators waiting. The same logic appears in the standalone tokenizer/detokenizer process and in the streaming shortcut that returns completed output without routing it back through every internal layer.[1][2]

Around 11:04, the relational tensor cache becomes the center of the design. It indexes reusable prompt prefixes and coordinates tensors across NPU high-bandwidth memory, host DRAM, SSD, and other serving engines. Crucially, a cache hit is not automatically a win. FlowServe uses a cost model to ask whether fetching preserved tensors will take less time than recomputing them, then performs a worthwhile transfer asynchronously. While one batch runs, the scheduler prepares the next; that keeps CPU scheduling and data movement from becoming bubbles on the NPU’s timeline.[1][2]

The tradeoff surfaces at about 12:48. Scheduling one step ahead means the stop condition can arrive after the next token has already been calculated, so the system may discard an extra token rather than stall the accelerator while it waits for the preceding result. That tiny example captures FlowServe’s larger policy: do speculative control work when the cost of occasionally throwing it away is lower than the cost of idling the scarce device.[1][2]

The paper’s performance plots should remain inside their evaluation boundary. Its offline FlowServe comparison uses a 34-billion-parameter model with tensor parallelism of four, 2K- or 4K-token inputs, and 256 decode steps. Its online prefill/decode comparison uses an internal trace with roughly 2K input tokens and 200 output tokens. These results support the mechanisms on the reported Ascend setup; they are not portable multipliers for every model, GPU, prompt mix, or latency target.[2]

13:15–15:17 — routing has no universal winner, and “cold” is a stack of delays

Hu accelerates through scheduling, but the compressed section contains the talk’s most important admission: there is no clean winner between colocating prefill and decode and separating them. The better choice changes with prompt length, expected output length, load, and interference. DeepServe first considers the worker type, then cache locality when loads are balanced, and finally the least-loaded option when imbalance becomes more costly than reuse. The paper’s decode-length predictor reaches 84.9% accuracy only after reducing the problem to 128-token buckets; uncertainty remains part of the scheduling problem, not something the architecture eliminates.[1][2]

At about 14:08, the talk decomposes scale-out into environment preparation, engine initialization, model loading, post-load setup, and cluster announcement. The first two steps can exceed ten minutes when Python libraries, NPU state, and cross-device communication groups all start from zero. DeepServe moves much of that work off the critical path with pre-warmed pods and model-agnostic task executors. It then either loads weights from a predicted model already present in host DRAM or copies them from a running NPU over a high-speed device link using what the paper calls NPU-fork.[1][2]

This is elasticity purchased with inventory. Warm pods consume capacity before demand arrives; DRAM preloading depends on predicting which model will need to grow; NPU-fork needs a live source instance and suitable interconnects. “Scale in seconds” is therefore not magic initialization. It is the result of deciding, ahead of time, which expensive steps to pay for and where reusable state should wait.

15:19 onward — the follow-up makes the hardware–software co-design explicit

The closing minute steps beyond the submitted paper. Hu previews a more deeply disaggregated system for large mixture-of-experts models, using DeepSeek-R1 as the example, and points toward a communication layer and serving design built for the SuperPod rather than merely adapted to it.[1] The subsequent xDeepServe report makes that trajectory concrete: it extends FlowServe across hundreds of NPUs, separates attention from feed-forward and expert computation, and introduces XCCL primitives over CloudMatrix384’s shared-memory fabric.[3]

Huawei’s later product messaging follows the same direction. At HUAWEI CONNECT in Shanghai on September 19, 2025, the company launched an AI Token Service powered by CloudMatrix384 and described the supernode as pooling compute, memory, and storage while disaggregating different kinds of work.[4] That announcement is commercial evidence of the platform strategy, not independent validation of every performance claim. It does, however, confirm why the research talk spends so much time on placement, movement, and modular execution: Huawei presents a token-level inference service as the layer customers consume, while its cloud absorbs the topology beneath it.

DeepServe’s most transferable lesson is therefore not an Ascend benchmark. It is a boundary. A good serverless interface lets a user ignore instances; a good serving system cannot. It must know where mutable state lives, when reuse beats recomputation, which phase of a request needs which worker, and how much warm capacity makes elasticity real. The less the caller sees, the more deliberately the provider has to see.

cronfeed.work