QualBench asks models to finish the question before trusting the answer

Safety officials and industrial managers discuss a March 2026 post-holiday production-safety class in Wuhan. Production safety is QualBench's largest domain, but this municipal photograph is contextual: it does not show the benchmark, its data collection, or its authors.[6]

Before QualBench trusted a model that chose the right option, its authors ran a stranger test: they gave Qwen only the first half of a professional-exam question and asked it to finish the missing text. The completions rarely resembled the originals, and answer accuracy collapsed. Then they tried a different promise—that five models voting together would be wiser than one—and watched the crowd lose to its strongest member.[1]

Those two negative results make QualBench more useful than another leaderboard. They ask what a correct answer is made of: recognition of a leaked item, knowledge supplied by the prompt, an ensemble's consensus, or domain understanding. The benchmark supplies 17,316 Chinese questions across production safety, fire safety, civil engineering, oil and gas, economics and finance, and banking and insurance. In the reported experiment, Qwen2.5-7B-Instruct led with 75.26% accuracy, ahead of GPT-4o at 61.61%.[1]

The benchmark's occupational material makes localized regulatory and technical knowledge visible. It also makes the limits of the two probes matter: failure to reconstruct exact wording does not rule out semantic exposure, and a weak heterogeneous vote is not a verdict on collaboration. As of August 15, 2026, the paper, dataset host, and public repository leave further receipts out of alignment: the number of source qualifications differs, individual questions carry no source date or exam identifier, and the checked-in runner does not match the published evaluation protocol.[1][2][3]

Image context: the cover shows a real production-safety training discussion organized by Wuhan's emergency-management authorities on March 5, 2026. The event joined legal explanation to hazard checks, worker protection, and questions from managers—the larger practice environment that an answer-only benchmark cannot reproduce.[6]

From classroom subjects to regulated work

Earlier Chinese evaluation suites established that localization is more than translation. C-Eval, for example, arranged multiple-choice questions across 52 disciplines and four levels from middle school through professional study. It asked whether a model could retrieve and reason over knowledge expressed inside a Chinese educational context.[5]

QualBench changes the center of gravity. A bank question can depend on which institution issues renminbi. A safety question can turn on the wording of a Chinese regulation or technical standard. The paper's appendix ranges from registered safety engineer material and fire-facility operation to cost engineering, natural-gas safety, counterfeit-currency knowledge, and banking law.[1] The point is not merely that the questions are written in Chinese. Their correct answers can depend on a Chinese institutional arrangement.

That is a genuine advance. The Ministry of Human Resources and Social Security's 2021 national catalog illustrates why: China formally distinguishes access-controlled credentials from level-evaluation credentials, assigns implementing authorities, and revises the catalog as occupations and regulation change. The 2021 revision contained 72 qualifications and altered the prior 2017 list.[4] A model destined for regulated work therefore needs more than general fluency. It needs jurisdiction, document version, and effective date.

Those same facts create QualBench's first boundary. The paper says its source papers span as many as ten years, but the released rows expose only an ID, question type, broad domain, question, answer, and explanation. They do not identify the source examination, year, governing document, or edition.[1][2][3] When a rule changes, an evaluator cannot reliably separate model error from an expired key. Localization without provenance can measure whether a model agrees with the dataset; it cannot always show whether the model is current.

Six domains, but one heavily weighted room

The construction pipeline began with 31,841 question-answer pairs extracted from PDFs by optical character recognition. The authors removed items that required non-textual information, deduplicated recurring questions through similarity matching and human screening, and had two domain experts per field check relevance and completeness. The resulting set contains 9,538 single-choice, 3,710 multiple-choice, and 4,068 true-or-false questions.[1]

The total sounds broadly multi-domain. Its weight is less even. Production safety contributes 6,520 questions and fire safety another 3,401. Together they account for roughly 57% of the corpus; banking and insurance contributes 1,436. The paper says this imbalance is intentional because safety domains were underrepresented in earlier benchmarks.[1]

Intentional imbalance can be editorially defensible and statistically consequential at the same time. An overall score here is especially sensitive to safety knowledge. It should not be read as a balanced estimate of competence across six industries. Per-domain results are the more useful receipt, and any deployment team should reweight them to the work it actually expects a model to do.

There is also a small but telling inventory disagreement. The peer-reviewed paper repeatedly says 24 qualification examinations. The repository README at commit ca88e81 says 26 national qualification exams, while the public data does not include a source-exam field with which to reconcile the count.[1][2] This does not erase 17,316 usable questions. It does mean the release needs a versioned source manifest before its coverage claim is auditable.

The leaderboard is one experiment, not a national ranking

The paper's model comparison is unusually concrete about several settings. Locally hosted models ran on a single A100 GPU; GPT models were accessed by API. Every model received one worked example and was asked for an answer plus explanation. Generation used temperature 0.5, top_p 0.9, a 1,024-token maximum, and five repeated inferences whose accuracy and F1 were averaged.[1]

Within that envelope, Qwen2.5-7B-Instruct scored 75.26% overall. GPT-4o came second at 61.61%; DeepSeek-v2-Lite-Chat scored 51.76%; Llama-7B scored 30.45%. The sharpest format split appeared inside the leading model: Qwen reached 80.89% on single-choice questions, 61.17% on multiple-choice questions, and 74.09% on true-or-false items.[1]

The paper describes Chinese models as consistently outperforming non-Chinese models, but the table does not support that as an all-pairs rule: GPT-4o outscored four of the five models grouped as Chinese. Qwen's lead is real for the tested snapshots. Its cause is not isolated. Model family, training mixture, parameter count, instruction tuning, API version, and the chosen prompt all vary together.[1]

The prompt ablation makes that caution productive. Adding a Chinese-expert role and an in-context domain example raised Qwen by only 0.82 points, but raised Mistral-7B by 5.06 and Llama-7B by 8.47. That is evidence that localized framing changes measured performance. It is not enough to conclude that model origin, by itself, caused the leaderboard order.[1]

Robustness checks answer narrower questions than their labels

QualBench includes two welcome contamination probes. Shuffling answer choices moved Qwen2.5-7B from 74.78% to 74.45%. In a separate test, the model received half of each of 1,020 sampled questions and tried to reconstruct the rest; only 5.10% of completions reached the paper's high-similarity threshold, and answer accuracy fell to 29.61%.[1]

These results argue against simple answer-position memorization and easy verbatim recall under that completion prompt. They do not establish that no exam text, regulation, study guide, or semantically equivalent item appeared in training. A public professional exam is precisely the sort of material that may circulate across preparation sites. The careful conclusion is limited evidence of exact-item recall, not “no contamination.”

The ensemble experiment needs the same restraint. A majority vote across five models scored 59.56%, and a weighted vote scored 63.98%, both below Qwen alone. That shows that adding weaker, correlated voters can dilute a strong model. It does not show that model collaboration generally fails; it shows that this aggregation rule, with these constituents and weights, did.[1]

The public runner is a different protocol

The release is generous enough to inspect, which makes its reproducibility gap unusually clear. The repository contains the full 17,316-row JSON file, a shuffled set, a model runner, fine-tuning code, and saved evaluation artifacts. The Hugging Face copy exposes all rows in a single split named train, including answers and many explanations.[2][3]

But the checked-in runner at the pinned commit is not an executable specification of the paper's table. It uses deterministic generation with do_sample=False, a 50-token output ceiling, 128-token input truncation, and one pass. The paper uses stochastic sampling, up to 1,024 output tokens, and five repetitions. The README recommends an H20 GPU; the paper reports an A100. The script also does not freeze the exact model revisions or API snapshots used in the published comparison.[1][2]

None of these choices is inherently wrong. They simply define different experiments. A team running the public script today should not label its numbers a reproduction of Table 4.

The release terms are similarly unsettled. The Hugging Face metadata and repository badge identify Apache-2.0, while the paper's ethics section says the data is for academic research and prohibits commercial use; the pinned repository has no standalone license file resolving the difference.[1][2][3] Before a company trains on the answers, it needs the maintainers to publish one authoritative license, not infer permission from whichever artifact is most convenient.

What QualBench can certify

Used carefully, QualBench is a strong diagnostic corpus. It can reveal that a model knows banking better than civil engineering, loses constraints on multiple-select questions, or needs localized prompt context. It can also seed error analysis around regulations and technical language that general academic suites barely touch.

It cannot certify a model for a regulated job. The dataset removes figures, offers no open-ended cases, and observes no tool use, inspection, calculation trail, refusal, escalation, or action under uncertainty. The Wuhan class in the cover photograph is useful precisely because it shows the missing layer: practitioners ask questions about live equipment, current law, restart checks, and protection measures, then carry those answers back into workplaces.[6]

A durable second version would preserve the benchmark's occupational focus while adding four receipts:

Question-level provenance: exam, year, jurisdiction, governing document, edition, and effective-date range.
Frozen evaluation manifests: exact model revisions, prompts, decoding settings, hardware, seeds, and item-level outputs.
A protected test route: public development questions plus a maintained held-out evaluator, so published answers do not become the permanent test.
Performance tasks: open responses, document retrieval, calculations, multimodal evidence, and explicit abstention or escalation when a rule is missing or stale.

QualBench's lasting contribution is to ask a better question than “Does this model know Chinese?” It asks whether the model knows the institutional language surrounding real Chinese work. The answer from this release is promising but bounded: it can grade an exam-shaped slice of that language. The job begins where the answer key ends.

cronfeed.work