A language model can miss a fact test in two observably different ways. Its response can omit the reference answer without contradicting it, or it can supply an answer that contradicts the reference. A leaderboard that records only the number of correct responses blurs that distinction. Chinese SimpleQA is valuable because it keeps the distinction visible.

Researchers affiliated with Alibaba Group first posted the benchmark in November 2024; it appeared as a long paper at ACL in July 2025. Its 3,000 short, open-ended questions cover six broad topics and 99 subtopics in Chinese. Instead of asking a model to select one of four supplied options, it asks for a fact and classifies the reply as correct, incorrect, or not attempted.[1][2]

That makes Chinese SimpleQA a useful instrument in 2026, but not a current model ranking. The published comparison uses 2024-era model aliases, including o1-preview, Doubao-pro-32k, GLM-4-Plus, GPT-4o, Qwen2.5, DeepSeek-V2.5 and InternLM2.5, as they behaved when run by the authors. Its lasting contribution is the shape of the test: a cheap way to examine factual recall, answer coverage, and stated confidence in Chinese. Its limits are equally important. Search access changes the task; a judge model affects the label; a public static set can enter later training data; and short answers say little about whether a long response remains factual from beginning to end.[1]

The unit of measurement is a response

Chinese SimpleQA inherits its basic design from OpenAI's English-language SimpleQA, which reduced factuality evaluation to short questions with one intended, stable answer. The smaller output space makes grading tractable: checking a name, place, title, or number is easier than auditing every claim in a page of prose.[3]

The Chinese benchmark adds a locally constructed subject map. Its questions span Chinese Culture; Humanities; Engineering, Technology, and Applied Sciences; Life, Art, and Culture; Society; and Natural Science. Each question had to be answerable by December 31, 2023, and its reference answer had to remain unchanged over time, rather than turn on a current officeholder, price, or live event.[1] That cutoff is part of the evaluation boundary. A correct score means recall on this deliberately timeless slice; it does not mean that a model knows what happened yesterday.

The scoring system preserves five views of the same run:

No one measure is enough. CGA can rise when more outputs are graded not attempted, while a higher attempt rate can coincide with more incorrect responses. The F-score makes an extreme not-attempted rate less attractive, but operators should still retain the underlying three-way counts. Two systems with similar F-scores can impose different verification costs if one produces more non-answers and the other more contradictions.[1] “Not attempted” is an output category, not direct evidence that a model consciously refused or knew it was uncertain.

The 2024 result was about response distribution as much as recall

The paper's setup says it evaluated 41 models—17 closed and 24 open—but its main results table contains 42 model rows: QwQ-32B-Preview appears in the table while being omitted from the enumerated open-model list. Using each provider's official or default sampling settings and GPT-4o as the main answer judge, o1-preview answered 63.8% of questions correctly; 12.2% of its responses were graded not attempted and 24.0% incorrect. Its correct-given-attempted score was 72.7%. Doubao-pro-32k reached 61.9% correct, 10.3% not attempted, and 27.8% incorrect.[1]

GPT-4o illustrates why the decomposition matters. It scored 59.3% correct—close to the two leaders—but only 1.4% of its responses were graded not attempted and 39.3% were incorrect. Claude 3.5 Sonnet was correct less often, at 46.2%, while 27.4% of its responses were graded not attempted and 26.4% incorrect. Those figures do not reveal either model's internal confidence or intent. They reveal response distributions that a single accuracy number would hide.[1]

The calibration experiment sharpened the point. Models were asked to attach a confidence score from 0 to 100 to each answer. For all eight systems plotted in the paper's calibration figure, binned empirical accuracy lay below stated confidence in bins above 50%; larger variants tended to be better calibrated than smaller relatives. This is aggregate evidence of overconfidence in that setup, not a reading of what any model “knew” on an individual question.[1]

These numbers should remain dated. They describe 2024-era provider aliases, prompts, defaults, and a GPT-4o judge—not the current Doubao, Qwen, DeepSeek, GLM, GPT, or Claude lines, and the paper does not consistently pin exact hosted snapshots. Re-running the set on a 2026 API without pinning the model version, judge, system prompt, sampling policy, and tool access would create a new experiment, not extend the old table.

Chinese knowledge changes the order

Chinese SimpleQA is not merely the English set translated. The team generated and verified a new corpus, and the published rankings changed accordingly. Within the paper's 12-model cross-benchmark comparison, Doubao-pro-32k moved from twelfth place on English SimpleQA to second on Chinese SimpleQA. In the Chinese Culture category, Doubao's reported F-score was 61.8, compared with 45.7 for o1-preview.[1]

That reversal is a feature, not evidence that either benchmark found a universally superior model. It shows that “factuality” depends on the language of the query, the distribution of facts, the names and transliterations accepted as answers, the model's training corpus, and its post-training behavior. English recall cannot stand in for Chinese recall. Nor can a Chinese-culture slice stand in for medicine, engineering, or live search.

Open-ended answers also remove the clues built into multiple choice. The paper reports a separate experiment converting a subset of C-Eval questions into open responses; performance fell when models could no longer recognize or guess among supplied options.[1] That makes the format a cleaner recall test. It still does not make it a reasoning test: the benchmark intentionally favors compact facts over multi-step explanation.

Retrieval does not improve the same capability—it changes the task

The paper's most operationally important experiment adds retrieval. With Google search results supplied through a LlamaIndex-based pipeline, GPT-4o's F-score rose from 59.7 to 81.8. Qwen2.5-3B rose from 17.3 to 72.5. The gap between the small and large Qwen2.5 variants compressed sharply once both could consult external material.[1]

A second run with Baidu results produced a lower F-score for all six systems in that comparison. Qwen2.5-3B reached 68.2 rather than 72.5; the authors attributed the difference to noisier retrieved context in their setup. The relevant lesson is not that one search engine is permanently better. It is that a retrieval score belongs to a complete system: query rewriting, index coverage, ranking, snippets, context packing, model instructions, and answer grading all contribute.[1]

Closed-book Chinese SimpleQA therefore measures short-form parametric recall plus observed correct, incorrect, and not-attempted rates. A retrieval-augmented run measures grounded answer production through a particular search stack. Both are useful, but placing them in one undifferentiated leaderboard would reward an undisclosed tool advantage. A production evaluation should report at least three lanes separately: no tools, a frozen common corpus, and live provider-specific retrieval.

Strong construction is not the same as an independent audit

The dataset was not casually scraped. The team began with 10,000 generated question-answer pairs, used model and retrieval checks, removed easy questions that five strong models could all answer, and retained roughly 3,000 after human review. Two annotators were required to check each candidate independently and supply at least two supporting URLs; a third adjudicated disagreements. In a separate robustness check, four Qwen2.5 judges and GPT-4o graded six selected systems: absolute scores differed, but their relative order remained stable.[1]

Those safeguards make the benchmark worth using. They also expose dependencies that a future audit should inspect. GPT-4o-0806 participated in question generation and automated validation, while GPT-4o served as the main judge. Difficulty filtering removed questions that all five named models answered correctly. A separate safety-risk filter removed questions and answers before release, but the paper does not publish a category-level rejection audit. None of this invalidates the set; it means the finished 3,000 questions reflect a production pipeline, not a neutral sample of all Chinese facts.[1]

Public availability creates a second boundary. The team checked new questions against its database before release and rewrote close matches to reduce direct memorization at construction time.[1] Since release, however, the full dataset has been downloadable from Hugging Face.[2] A later model may encounter those exact questions during pre-training, supervised tuning, evaluation tuning, or synthetic-data generation. “Static” keeps the reference answer stable; it does not keep the test secret.

There is a useful warning from the benchmark's English ancestor. In 2025, Google researchers produced SimpleQA Verified after finding noisy labels, topic imbalance, and redundant questions in the original English set. Their work reconciled sources, rebalanced topics, deduplicated questions, and revised the autorater.[4] That is not evidence that Chinese SimpleQA has the same defects. It is evidence that short, stable answers do not eliminate the need for independent item-level review.

Chinese SimpleQA's own limitations section is appropriately narrow: six topics cannot cover every niche; a static set misses evolving facts; and short-form scoring overlooks complex reasoning and nuanced factuality.[1] A credible 2026 refresh would add a hidden, independently authored split; publish exclusion and adjudication statistics by category; test multiple Chinese-capable judges against human labels; and measure overlap with common training corpora.

How to read a Chinese SimpleQA score in 2026

A useful scorecard should answer five questions before presenting the rank:

  1. Which artifact ran? Pin the exact model snapshot, endpoint region, prompt, sampling settings, and judge version.
  2. How did it fail? Publish correct, incorrect, not-attempted, correct-given-attempted, and calibration—not only F-score.
  3. What could it see? Separate closed-book, frozen-retrieval, and live-search results; identify the retrieval provider and corpus date.
  4. Where did it work? Break out Chinese culture and the other broad topics so aggregate gains do not conceal a regional or domain deficit.
  5. Could it have seen the exam? Pair the public set with a fresh hidden set and investigate suspiciously large gains item by item.

Used this way, Chinese SimpleQA is more durable than its original leaderboard. It does not certify that a model is truthful, and it cannot simulate the factual burden of a long report or an agent acting on retrieved information. It does something smaller and unusually legible: it separates correct responses, contradictions, and responses that omit the reference answer. That distinction—not first place—is the result worth preserving.

Sources

  1. Yancheng He et al., “Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models,” Proceedings of ACL 2025—dataset construction, stated 41-model setup and 42-row results table, metrics, calibration, retrieval experiments, judge analysis, and limitations.
  2. OpenStellarTeam, Chinese-SimpleQA on Hugging Face—public dataset card, 3,000-question corpus, subject coverage, files, and paper linkage.
  3. Jason Wei et al., “Measuring short-form factuality in large language models” (November 2024)—the original English SimpleQA design and correct/incorrect/not-attempted evaluation frame.
  4. Lukas Haas et al., “SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge” (first posted September 9, 2025; revised March 10, 2026)—independent source reconciliation, deduplication, topic balancing, and autorater revision for the English benchmark.
  5. Alibaba Cloud Community, “Green by Design: Inside Alibaba's International Headquarters Built for Sustainability” (June 5, 2024)—official source page for the documentary photograph of Alibaba's Xixi global headquarters used as the article image.