Wu Dao’s trillion-parameter headline hid a model system

Jie Tang presents at the Wu Dao 1.0 launch in March 2021, three months before Wu Dao 2.0’s debut. This Tsinghua University photograph records the program’s public origin; the screen is part of the photographed stage, not an analytical graphic made for this article.[7]

Five years after Wu Dao 2.0 arrived, its easiest fact to remember is still its least explanatory one: 1.75 trillion parameters. In June 2021, the Beijing Academy of Artificial Intelligence presented that figure as a world record and as ten times GPT-3’s parameter count. The same announcement described something broader, however—a bilingual, multimodal “model system” supported by training frameworks, Chinese corpora, applications, and an alliance of 22 partner organizations.[2]

Jie Tang’s ECML PKDD 2021 keynote is valuable because it preserves both versions of the story. It has the period’s spectacle: machine-written Chinese poetry, text-to-image examples, a virtual student, and the giant number. It also reveals the engineering program underneath, then ends with unusually direct admissions about slow inference, incorrect fluent answers, and knowledge that becomes stale inside model weights.[1]

The 54-minute English-language recording was published by the conference’s official channel. Some demonstrations and slides are in Chinese, but the argument is spoken in English. Watch for the moment the presentation changes scale: the opening demos ask what one system might appear to do; the middle explains the different models, data, and training mechanisms needed to do it; the questions at the end expose what none of those components had solved. The timestamps below are landmarks rather than a transcript.[1]

7:49–16:33 — the demo reel makes one system look like one model

Tang begins with outputs rather than infrastructure. Wu Dao answers questions, writes prose in a requested persona, composes Chinese poetry, captions images, and turns prompts into pictures. Around 10:25, a young polar bear, a bus, and a football-playing tiger demonstrate text-to-image generation; around 13:58, the story shifts from playful examples to personalized product design for e-commerce. By about 14:58, a public test asks visitors to distinguish human work from generated poems and captions.[1]

This sequence is effective stagecraft, but it encourages a category mistake. A viewer can easily imagine one enormous network moving seamlessly from poetry to pictures. The research record points instead to a coordinated family. CogView, for example, was a four-billion-parameter Transformer trained for text-to-image generation, with its own architecture, data pipeline, and evaluation against other image-generation systems.[6] Tsinghua’s 2021 research feature likewise described Wu Dao through several branches—language generation, multimodal work, general language modelling, and protein prediction—rather than as one chat model with every faculty folded into it.[7]

That distinction does not diminish the program. It makes the achievement more legible. “Wu Dao” functioned as an umbrella for a model portfolio and the infrastructure around it. The keynote’s variety is evidence of coordinated research scope, not proof that every example came from the same checkpoint under the same inference conditions.

16:33–19:41 — the record number is an infrastructure claim

At 16:33, Tang finally places 1.75 trillion on screen. He connects the result to bilingual text and image data, a Sunway supercomputer, software written for that machine, distributed communication, and FastMoE. The official launch account gives the data total as 4.9 terabytes: 1.2 TB of Chinese text, 2.5 TB of Chinese image–text data, and 1.2 TB of English text from the Pile.[2]

The related WuDaoCorpora paper makes the data layer more concrete. It documents a three-terabyte Chinese corpus assembled from 822 million web pages, while distinguishing that full collection from a roughly 200 GB base version made available for research. Its published baseline was a three-billion-parameter Transformer-XL trained on the base subset—a much narrower experiment than the entire Wu Dao 2.0 claim.[3] “The data exists,” “the data is released,” and “a named model was evaluated on it” are three separate statements.

FastMoE supplies the other boundary. Mixture-of-experts systems expand total capacity by distributing experts across devices and routing work among them; all stored parameters are not used as one dense block for every input. The FastMoE paper presents this primarily as an algorithm-and-systems problem: how to place experts across ordinary GPU nodes, communicate efficiently, and keep a PyTorch-based training system usable.[4] The trillion-parameter figure therefore demonstrates sparse capacity and distributed execution. It does not, by itself, show ten times GPT-3’s computation per token or ten times its capability. Neither the keynote nor the launch notice provides a shared evaluation envelope that could support that conversion.[1][2][4]

The more durable reading is physical. Wu Dao’s headline could only exist because corpus construction, sparse routing, accelerator topology, communication software, and model objectives were developed together. The number is the visible edge of that stack.

19:41–35:09 — “general” meant unifying objectives, not erasing components

Around 19:41, the talk turns from scale to GLM, the General Language Model framework. Tang contrasts autoregressive models suited to open-ended continuation, autoencoding models suited to understanding tasks, and encoder–decoder models used for conditional generation. The proposed answer is autoregressive blank infilling: blank out spans, then generate their contents in a way that can be adapted to understanding, conditional generation, or unconditional generation.[1]

The peer-reviewed GLM paper later specified the mechanism and its evaluation boundary. It used two-dimensional positional encodings and variable blank lengths and orders, then compared models of matched scale and data across the three task families. Its claim was not that the 1.75-trillion-parameter Wu Dao checkpoint won every task. It was that one pretraining framework could perform competitively across categories that had usually favored different architectures.[5]

That is a subtler kind of generality than the demo reel implies. The objective may be reusable while the deployed models, modalities, sizes, and downstream tuning remain distinct. The talk itself confirms this layered design. It moves from GLM to parameter-efficient adaptation, then to a separate generation method that scores whether each new sentence remains connected to the original prompt. Meanwhile, the image examples rest on CogView rather than on GLM alone.[1][5][6]

For a modern viewer, this section is a useful antidote to model-family marketing. A common research spine can reduce duplication without making every checkpoint interchangeable. Architecture, data, modality, serving cost, and evaluation setup still define which capability is actually present.

41:22–44:27 — the candid limitations outlast the record

The most forward-looking part arrives in the questions. Asked what the system cannot do, Tang names two limits. First, the largest model is expensive enough that an answer may take minutes, motivating smaller distilled models and more efficient inference. Second, generated text can look fluent while being factually wrong. A follow-up asks how knowledge encoded in weights can be updated; Tang describes lifelong-learning research but says the practical process still requires periodically retrieving data and rebuilding or combining models.[1]

Those answers redraw the whole presentation. The demo section measures whether an output resembles something a person might write or draw. The Q&A asks whether the output arrives cheaply, whether it is true, and whether its knowledge can change. These are not cleanup details after scale. They determine whether a foundation model can become dependable infrastructure.

The launch announcement imagined large models as something like a power station supplying intelligence to an application network.[2] The analogy leaves out the difficult meter. Electricity is fungible at the socket; model output is conditional on a version, prompt, corpus, route, and evaluation. A useful intelligence service needs provenance and quality boundaries, not only access. Wu Dao’s own presentation supplies the reasons: the biggest checkpoint was costly to serve, fluent generation was not guaranteed factual, and knowledge refresh remained unfinished.[1]

What the recording preserves

Wu Dao 2.0 deserves a place in AI-China history, but not because its parameter record settled a race. The keynote captures a Chinese research coalition assembling several pieces of the foundation-model era at once: a much larger native-language corpus, sparse distributed training, a general language objective, multimodal generation, adaptation methods, and a planned developer ecosystem.[2][3][4][5][6]

It also preserves the gap between a program and a product. The public name made many models and tools feel singular; the technical discussion made their boundaries visible. Rewatching the talk now, the right question is not “How big was Wu Dao?” It is “Which layer produced each result, under what conditions, and what remained unresolved?” The record number has aged. That method of reading has not.

cronfeed.work