Qwen's million-token window has a 2,048-token keyhole

Alibaba's booth at the World Artificial Intelligence Conference in Shanghai in July 2025. The crowded, physical setting is a useful counterpoint to a million-token model claim: what matters is not the size of the sign, but how the system chooses what to inspect. Costfoto/NurPhoto via Getty Images.[7]

The headline number on Qwen3.8-Flash-Next is a one-million-token context window. The more revealing number is 2,048. At each sparse-attention query, Alibaba's new open-weight model uses a learned indexer to choose a budget of up to 2,048 earlier tokens, grouped into as many as 512 four-token micro-blocks, plus the unfinished block at the sequence tail. It does not repeatedly apply dense attention to every token in the million-token window.[1][2]

That distinction is the release's real contribution. Qwen has not merely attached a search filter to a conventional transformer. It trained an indexer to imitate dense attention, then trained the backbone to work with the indexer's sparse choices. In the team's own tests, this learned sparsity is faster at the attention-kernel level and sometimes more accurate than the dense baseline on long-context retrieval. Yet the same tables also show how far “supports one million tokens” remains from “understands anything placed inside one million tokens.”[2]

The result deserves attention because it is unusually inspectable. The weights are public on Hugging Face and China's ModelScope, the technical report includes the training sequence and ablations, and the repository supplies launch paths for SGLang, vLLM, Transformers, and other runtimes.[3][4] It also deserves a narrow reading: nearly all capability evidence currently comes from Qwen's own evaluation package.

A million-token window with a selective view

Qwen3.8-Flash-Next alternates three Gated DeltaNet layers with one Qwen Sparse Attention layer. The Gated DeltaNet layers continually compress the prefix into a fixed-size recurrent state. The QSA layer restores direct retrieval, but only after its lightweight indexer scores compressed blocks and selects the most relevant regions. This division gives the model two kinds of memory: a running summary for continuity and a selective route back to particular tokens.[1][2]

The released model has a 125-billion-parameter main network, with about 6 billion parameters activated per token, plus 51 billion parameters in an N-gram embedding table held off the accelerator. Its native context is 262,144 tokens; Qwen extends that to one million with YaRN, while the managed qwen3.8-flash service exposes one million tokens by default.[1][6] Those details matter because the long-window claim combines several mechanisms. Sparse attention limits direct retrieval work, the recurrent layers avoid a dense key-value cache at most layers, and the hosted product chooses its own serving configuration.

Within each QSA layer, the indexer uses four query heads and one shared key head. It compresses keys four at a time, ranks those micro-blocks, expands the winners back to token positions, and applies core attention only there. The 2,048-token selection budget is constant even as the available context grows. That is the economic bargain: a longer shelf of documents without asking every query to reread the whole shelf.[2]

It is also a semantic gamble. If the indexer fails to rank a needed passage, the direct-attention layer cannot recover it from an unselected block. The recurrent Gated DeltaNet state may preserve a useful trace, but it is finite and compressed. Long-context quality therefore depends not only on how much text fits through the API, but on whether the learned selection policy recognizes the evidence a particular query will need.

The indexer is trained, not bolted on

The technical report's most consequential finding is easy to miss: applying the indexer directly after dense initialization caused a clear performance drop. Qwen recovered by making sparse attention part of continued pretraining.[2]

First came dense distillation. For 1,000 steps, each using eight sequences of 256,000 tokens, the indexer learned from the full-attention distribution while the rest of the model stayed fixed. Qwen estimates that warm-up at roughly 2 billion tokens. The teacher signal was pooled from token-level attention into blocks, teaching the smaller indexer where dense attention would have looked.[2]

Then the model crossed the important boundary. QSA selected the blocks used by core attention, and both indexer and backbone trained together for another 8,000 steps. Each step used 96 sequences of 256,000 tokens, for roughly 200 billion tokens of sparse training. The backbone was no longer being asked to tolerate a post-hoc shortcut; it was adapting its representations to the shortcut.[2]

This changes the engineering lesson. QSA is not a generic runtime switch that an operator should expect to apply to any dense model without loss. Its selection behavior, weights, training curriculum, fused kernel, and backbone are a package. The open repository's Day 0 support across multiple engines is valuable, but support means faithfully executing that package—not inventing equivalent sparsity around an unrelated checkpoint.[3]

The benchmark win contains its own warning

On eight general benchmarks spanning knowledge, mathematics, reasoning, multilingual performance, and code, the QSA version averaged 76.8, versus 75.9 for Qwen's full-attention baseline. It matched or led on seven of eight tests. That is useful evidence against a simple fear that sparsity must degrade ordinary capability.[2]

The long-context tables are more revealing. On RULER, averaged over the 512,000-to-one-million-token range, QSA scored 93.00 against 90.08 for full attention. On eight-needle MRCR, QSA improved the 512,000-token result from 30.66 to 40.53, and the one-million-token result from 20.71 to 26.44.[2]

Both readings should remain visible. QSA beat the team's dense baseline as context became very long, so the selector did more than preserve speed. Joint training may have helped it focus retrieval on high-value regions. But MRCR's absolute one-million-token score remained low: QSA matched the benchmark's requested target better than full attention under this setup, yet remained far from a perfect result. A context window can be technically addressable while difficult evidence is still missed.

RULER and MRCR are retrieval-oriented tests. They are well suited to checking whether distant material survives, but they do not establish reliable reasoning over an entire repository, a year of financial records, or a mixed collection of text and images. Qwen's public release blog adds coding, office-work, tool-use, and multimodal scores, yet those comparisons introduce different harnesses, prompts, graders, and competitor configurations.[1] The clean claim is therefore local: in Qwen's paired ablation, trained sparse attention did not trade away the tested short-context capability and improved the reported long-context retrieval average.

Three speed claims, three different experiments

The report measures QSA's efficiency at the attention-module level. At one million tokens, it reports a 7.6× prefill speedup and a 4.9× decode speedup over paged grouped-query attention. The setup is specific: prefill processes the last 16,000-token chunk at batch size one; decode uses batch size four with three extra multi-token-prediction steps. The measurement includes the QSA indexer and sparse core attention.[2]

It does not include every cost of serving the model. Mixture-of-experts routing, feed-forward layers, the widened residual stream, N-gram memory fetches, vision processing, tokenization, request queues, networking, and an agent's tool loop all sit outside that kernel comparison. The multipliers are credible evidence that attention becomes cheaper under the published setup, not a promise that every application will finish 7.6 times faster.

Nvidia offers a second experiment. Its Day 0 test on a GB300 NVL72 reports more than 16,000 tokens per second per GPU and more than 200 tokens per second per user. That platform links 72 Blackwell Ultra GPUs in one NVLink domain with 130 terabytes per second of aggregate bandwidth.[5] The result shows that the checkpoint can be made fast on a rack-scale system. The post does not disclose enough workload detail to equate its numbers with Qwen's one-million-token kernel test, and such a system says little about smaller clusters.

The managed QwenCloud release is a third surface. It makes the model available as qwen3.8-flash, with a one-million-token context window and compatible API protocols.[6] That proves product integration, not the latency or retrieval quality of an arbitrary million-token request. Kernel, rack, and API measurements answer different questions; merging them would create a performance claim none of the sources makes.

What a serious evaluation should preserve

For builders, the right test is not “does a one-million-token request return?” It has three parts.

First, measure selection survival. Hide necessary evidence at varied positions in real documents, include plausible distractors, change the wording of the final query, and record accuracy as context grows. A selector that finds exact repeated strings may still miss a requirement expressed through paraphrase or spread across several blocks.

Second, separate attention speed from service speed. Record time to first token, inter-token latency, total completion time, throughput, and tail latency at the batch sizes and prefix-cache hit rates the application will actually see. If kernel gains disappear behind expert communication or memory traffic, the deployment result—not the architecture diagram—sets the budget.

Third, test the supported implementation, not an imagined generic version of the idea. Qwen's repository names specific engines and launch configurations; ModelScope supplies a Chinese first-hand distribution route for the same released model.[3][4] Version, precision, tensor parallelism, context extension, and multimodal inputs all belong in the test record.

The claim has a straightforward falsifier. If independent, matched end-to-end tests show that QSA's retrieval advantage vanishes on realistic corpora, or that indexer and system overhead erase the speed gain outside the attention kernel, then this is a specialized optimization rather than a new long-context default. If the paired advantage survives, Qwen will have demonstrated something more important than a large context label: a learned sparse route can outperform exhaustive attention on the work it was trained to find.

That is the durable AI-China signal. The contest is moving from publishing ever-larger windows to publishing the mechanism that makes those windows usable—and enough ablation detail for outsiders to discover where the mechanism still fails.

cronfeed.work