Llumnix moves a live LLM request instead of waiting for the queue

Alibaba Cloud's exhibition floor at the 2024 Apsara Conference. The photograph provides company and infrastructure context; it does not depict the Llumnix experiment or the OSDI talk. Source: www.alibabagroup.com.[6]

An LLM request changes shape after it has been assigned to a GPU. Its prompt is visible at arrival, but its output length is not. Every generated token extends the key-value cache, so a request that looked cheap at dispatch can become the memory-heavy neighbor that slows a whole batch. A one-time routing decision is therefore being asked to predict information that autoregressive generation has not revealed yet.

That is the problem Alibaba engineers Biao Sun, Ziming Huang, Hanyu Zhao and colleagues bring to the 2024 USENIX OSDI stage. Their system, Llumnix, adds continuous rescheduling above multiple vLLM instances. Its central move is unusually literal: if the original placement turns out to be poor, migrate the running request and its KV-cache state instead of merely accepting the queue or recomputing the request elsewhere.[1][2]

The roughly 16-minute talk is compact enough for two passes. On the first, follow the system in five viewing checkpoints. On the second, keep three columns—claim, mechanism, boundary—and refuse to put a result in the claim column until the mechanism and test conditions are also visible. The time ranges below are navigation guides, not a transcript.[1][3]

0:33–3:25 — a dispatcher makes an irreversible guess

The opening diagram looks ordinary: users send requests to a dispatcher, which spreads them among several model instances. The inference engine inside each instance may already use continuous batching, paged KV-cache allocation and preemption. Llumnix's complaint is not that those local techniques are useless. It is that cluster dispatch inherited a traditional deep-learning assumption: requests are fairly homogeneous, deterministic and stateless. LLM requests violate all three parts.[2][3]

Watch how the talk separates input length from output length. The first is known at arrival; the second emerges token by token. That uncertainty makes GPU memory demand dynamic because the KV cache grows with the generated sequence. It also means equal request counts are not equal loads. Two instances can each receive ten requests and end up with very different memory pressure, queueing and interference.

3:25–8:05 — rescheduling turns four cases into one capability

This is why the right unit of concern is tail latency, not just average throughput. A busy instance may preempt a running request, forcing a stall and possibly recomputation. A cluster may have enough free memory in total for a large prompt but not enough on any single instance, leaving the request at the head of a queue. And an urgent interactive request can share a batch with less time-sensitive work. From 3:25, the talk names these as performance isolation, fragmentation and differentiated service-level objectives—not as three separate products, but as consequences of a placement that cannot be revised.[2]

The operating-system analogy arrives around 6:02. A process has an unknown duration and a changing working set; an OS does not promise never to move or preempt it after launch. Llumnix applies that systems intuition to LLM serving. The comparison is useful if kept narrow: a request can be rescheduled like a process, but its state is a specialized tensor cache and its progress is autoregressive decoding, not a general CPU process.[1][2]

The slides show four reasons to move work. Load balancing reacts after actual cache growth reveals an overloaded instance. Defragmentation packs some running requests elsewhere to leave enough free KV-cache capacity on one instance for a queued prompt. Prioritization moves normal work away from a latency-sensitive request rather than reserving a machine permanently. Autoscaling drains a terminating instance or fills a new one faster. Dispatch still matters; migration gives the scheduler a second decision after the workload has disclosed more information.[3]

That distinction prevents a common overreading. Llumnix does not predict the final length of every answer. It reduces the penalty for being wrong. The strategic value lies in reversibility: observe real growth, then alter placement.

8:05–11:25 — copy the stable past while decoding the next token

Moving a request sounds expensive because its KV cache can be large. The naive choices make that cost visible to the user. Suspend the request and copy everything, and service pauses for the full transfer. Start it again on the destination and recompute its history, and the pause grows with the amount of prior context. Llumnix instead exploits a structural property: KV-cache blocks for earlier tokens are append-only. New decoding steps append state; they do not rewrite the old blocks.[2]

During the first migration stage, the source keeps decoding while completed cache blocks are copied to the destination. A later stage copies the smaller increment produced during that first transfer. Only at the end does Llumnix drain the request from the source batch, copy the final increment and resume it on the destination. The bulk transfer still takes time and consumes bandwidth. What the design tries to keep nearly constant is the downtime experienced by the migrated request, not the total number of bytes moved.[2][3]

The paper's controlled migration test makes that boundary concrete. On its LLaMA-7B and LLaMA-30B setups, reported downtime stayed around 20–30 milliseconds as sequence length increased, while recomputing an 8,000-token LLaMA-30B sequence took 3.5 seconds. The measured per-step decode difference for other running requests was up to 1%, and migrations were in progress for roughly 10% of each instance's time in the later serving experiments. Those are strong measurements for this testbed, not a law that every model, fabric or cache layout will reproduce.[2]

Turn from the recording to Figure 7 in the paper for the handshake omitted from the short deck. Before each stage, the destination reserves enough blocks; either side can abort if memory is unavailable, the request finishes, it is preempted or a participant fails. That control path is as important as the copying trick. “Live migration” is not just a fast tensor transfer. It is a coordinated change of ownership while computation continues.[2]

11:25–13:40 — virtual usage makes policy legible

Llumnix splits scheduling between a global scheduler and per-instance components called llumlets. The global layer sees instance loads, dispatches arrivals, pairs migration sources with destinations and controls scaling. A llumlet maintains local request state, chooses which request to move when its instance is selected, and coordinates the transfer. This keeps the global scheduler from tracking every running request in detail.[2][3]

The load it reports is not always physical memory. Llumnix introduces virtual usage, an accounting value that lets one load-balancing policy express different objectives. In the normal case, virtual usage follows actual memory use. For a blocked prompt, the scheduler can count the prompt's unmet memory demand so that moving running work creates room. A high-priority request can receive virtual headroom, making its instance appear fuller and pushing normal requests away. A terminating instance can be assigned artificial load so that work drains from it.[2]

This is the most reusable design idea in the talk because it distinguishes mechanism from policy. Live migration makes placement reversible; virtual usage tells the system when a placement is undesirable. Neither proves that the chosen heuristic is optimal. The paper itself describes a simple rule set and notes policy choices and trade-offs. A production operator would still have to decide how much migration traffic, headroom and queue preference the service can afford.

13:40–15:52 — read every speedup with its denominator

The presentation's evaluation slide gives the denominator before the result: 16 NVIDIA A10 GPUs with 24 GB each, arranged as four four-GPU Alibaba Cloud virtual machines connected by 64 Gb/s networking. The paper evaluates 16-bit LLaMA-7B and LLaMA-30B, with ShareGPT, BurstGPT and generated power-law request-length distributions. The serving comparison holds vLLM underneath each scheduler and contrasts Llumnix with round-robin dispatch and an authors' optimized INFaaS baseline called INFaaS++.[2][3]

In the real-trace experiments, Llumnix reduced mean and P99 first-token latency by as much as 2.2× and 5.5× relative to INFaaS++, and improved P99 per-token decode latency by up to 1.3×. Across generated distributions, the paper reports improvements up to 7.7× for mean and 14.8× for P99 first-token latency. Other experiments report high-priority latency gains up to 1.5× and 36% lower average instance cost at roughly similar P99 first-token latency under a selected autoscaling comparison.[2]

Those maxima do not belong in one imaginary universal workload. They come from different traces, arrival rates, objectives and comparisons. The slides deliberately use smaller headline numbers from the real datasets; the paper contains the broader maxima. A careful viewing records which baseline and metric sit under each “up to.” It also notes that the 64-instance scalability study replaced real GPU execution with timed simulation because that cluster exceeded the physical testbed.[2][3]

The defensible conclusion is narrower and more useful than “Llumnix makes inference 15 times faster.” In the tested multi-instance configurations, migration let the scheduler repair load imbalance and fragmentation after dispatch, substantially improving the long tail in conditions where one-shot placement was costly.

The 2026 repository is not the OSDI artifact frozen in time

The software boundary changed after the talk. The original Ray-based repository is now labeled Llumnix v0 and directs production-oriented users toward a re-architected, modular and cloud-native Llumnix v1 repository. The v0 page describes itself as the better option for local deployment and rapid scheduling experiments; it also labels that line alpha-stage and publishes later benchmarks on Qwen2.5-7B/A10 and Llama2-13B/A800 configurations. Those later round-robin and queue-size comparisons are not the OSDI experiment and should not be blended with its results.[5]

The v1 repository describes a larger serving stack: an initial scheduler plus rescheduler, full and engine-transparent modes, a gateway, instance-state tracking, KV-transfer components, fault handling, and support for vLLM and SGLang integration. It also says Llumnix has been used in Alibaba Cloud PAI-EAS and identifies the March 2026 release as a new architecture.[4] That is first-party project documentation, not an independent production audit. It shows where the maintainers took the idea; it does not retroactively widen the evidence in the 2024 paper.

Second pass — test whether migration earns its keep

On a second viewing, ask three operational questions. First, what state must cross the network? Longer contexts and larger models increase cache volume even if the final pause remains short. Second, what does migration contend with? Cache traffic shares a real fabric with inference, storage and other transfers; “near-zero downtime” is not “zero resource cost.” Third, what failure changes ownership? Reservation, abort and commit paths determine whether a fast transfer is also a correct one.

Then separate three kinds of proof. The OSDI work supplies a mechanism, a controlled testbed and comparative latency results.[2] The repositories supply evolving implementation claims and deployable code.[4][5] Neither alone supplies an operator's result on a new model, accelerator, topology or traffic distribution. That result needs a local trial measuring time to first token, time between tokens, preemption stalls, migration frequency, transferred bytes and failure recovery under the service's own workload.

The lasting AI-China signal is therefore not a single benchmark win. It is a shift in where inference performance is being engineered. Model weights and kernels matter, but so does the cluster's ability to revise a decision after an unpredictable request has begun. Llumnix's strongest lesson is simple: when the workload reveals itself over time, the scheduler should be allowed to learn from that revelation too.

cronfeed.work