Imagine answering a geometry question correctly, then being handed the same problem with its answer choices rearranged. The number you chose is still there. Its letter has changed. Would you follow the number, or repeat the letter?

That small disturbance sits at the heart of CMMU, a Chinese benchmark for models that read both images and text. In its January 27, 2024 announcement, the Beijing Academy of Artificial Intelligence described a test drawn from school examinations, with single-answer, multiple-response, and fill-in-the-blank questions. Its first release, v0.1, contained 3,603 questions across seven subjects.[1]

Revisiting that release reveals something more durable than a historical ranking. The examiner's choices determine what a correct answer demonstrates. CMMU asks whether an answer survives rearrangement, whether a selection is complete, and whether different wording deserves credit. Those are three different demands on a model—and three reasons to read the scoring procedure before comparing percentages.

Following the answer around the page

For a single-answer question with four options, CMMU cycles the choices through four positions. The model receives each arrangement separately. Credit requires getting every arrangement right. BAAI's release announcement calls this ShiftCheck; the IJCAI 2024 paper describes the underlying procedure as CircularEval.[1][2]

Consider an illustrative model that always returns B, regardless of the question. Across four rotations, it encounters the correct answer at B once. An evaluator averaging those four responses would award 25 percent. CMMU's question-level rule awards zero: the model failed to track the answer across all arrangements.

That distinction also affects more capable models. Three correct responses and one mistake still lose the point. My interpretation is that the resulting accuracy combines solving the problem with remaining reliable under a specific change of presentation. It should not be read as the same quantity as ordinary accuracy from one arrangement per question.

The paper separately measures how unevenly answer letters are selected among questions that fail the rotation test. Its Positional Error Variance is a diagnostic of position preference.[2] A model could nevertheless be consistently wrong without favoring any letter. Low position bias and strong subject knowledge therefore require separate evidence.

No consolation point for an incomplete selection

Multiple-response questions introduce another boundary. The released evaluator cleans the response, sorts its selected letters, and compares the result with the reference answer. The selection must match completely.[3]

Suppose the correct choices are A and C. Choosing A alone shows some success at recognizing a true statement, but earns no credit here. Choosing A, C, and D also fails. This is an illustrative example of the code's rule, rather than a reproduced dataset question. The score rewards identifying the complete correct set without adding a false choice.

For an evaluator, that creates a useful distinction between two errors hidden behind the same zero: missing something true and accepting something false. A diagnostic review could retain both categories alongside the official score. That additional analysis would help explain the failure without silently changing what the published metric measures.

Fill-in-the-blank questions remove the menu of answers. The repository's loader also breaks a question with several subquestions into separate records, carrying the shared question text into each one.[4] This makes the unit being counted consequential: a record can represent one component of a longer exercise. A percentage needs its denominator and preprocessing history, not merely the name of the exam collection.

Change the marker, change the mark

The sharpest demonstration appears in the repository's two results tables. With GPT-4 grading the written answers, the historical Qwen-VL-Plus test average is 27.73 percent. With rule-based grading of those answers, it is 21.34 percent. The difference is 6.39 percentage points, calculated from the reported values. These are overall test averages under the two grading methods, rather than scores for the fill-in-the-blank subset alone.[4]

The paper's original experiment used Chinese prompts in a zero-shot setting. It ran open models on NVIDIA A100 GPUs and accessed closed models through their APIs. Choice questions used temperature zero and a ten-token output allowance; written answers used temperature 0.2 and an allowance of 128 tokens.[2] The figures belong to that 2024 evaluation. They are not measurements of today's model versions.

The alternative marker is worth inspecting. In the released code, rule-based grading normalizes the reference and response, then checks whether a response line exceeds a string-similarity threshold of 0.7. The model-based route instead submits the question, reference, and response to a language-model judge.[3]

Neither operation is identical to a teacher establishing that a student understands an idea. Similar strings can carry different meanings; different strings can express the same answer. A model judge introduces its own judgment errors. The gap between the tables establishes that the marking method matters, but cannot by itself tell us which individual decisions were fairer.

What a comparable improvement would show

My proposed reading of a new CMMU result begins with continuity: the same question split, rotations, answer extraction, and written-answer judge, with the tested model and generation settings identified. If one of those changes, a useful report would show a separate comparison that isolates its effect.

I would also want the successes unpacked. Did the model follow the right answer through every position? Did it select every correct statement without adding another? Did it produce an answer a reviewer could defend independently of the automatic marker? These questions turn a leaderboard gain into something inspectable.

CMMU's lasting contribution is the demand embodied by its rotating options: being right once leaves an important question unanswered. Move the answer, preserve the problem, and see whether the model can still find it.

Sources

1. Beijing Academy of Artificial Intelligence, “BAAI releases the Chinese multimodal benchmark CMMU,” January 27, 2024 — Chinese-language first-hand release announcement, v0.1 scope, and ShiftCheck procedure.
2. Zheqi He et al., “CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning,” IJCAI 2024 — Sections 4–5, CircularEval, positional bias, and original experimental settings.
3. CMMU authors, released evaluator implementation — complete-match multiple-response scoring and the two fill-in-the-blank grading paths; accessed October 19, 2026.
4. CMMU authors, official repository README — subquestion conversion and separate results tables for GPT-4 and rule-based grading; accessed October 19, 2026.
5. Beijing Academy of Artificial Intelligence, report on the close of the 2024 BAAI Conference, June 16, 2024 — institutional source for the conference photograph.