Imagine giving an AI a photograph of a kitchen and asking which cupboard stands open. It chooses correctly. Now remove the photograph and repeat the question. Another correct answer would make the first success harder to interpret: perhaps the wording gives something away, perhaps the model remembers the example, or perhaps it guessed. To learn what the image contributed, the evaluator needs a comparison.

MMStar made that comparison central to a benchmark. Developed by researchers affiliated with the University of Science and Technology of China, the Chinese University of Hong Kong and Shanghai AI Laboratory, the 2024 project examines how much credit a visual model should receive for answering a visual question.[1] Its useful contribution survives changes in the leaderboard: it gives the picture a control condition.

Three runs behind one result

The official repository asks contributors to supply three result files: the vision-language model receiving images, that same model receiving only the textual questions and choices, and its original language-model backbone receiving text alone.[2]

Each run answers a different question. The first measures success on the complete task. The second asks what remains possible after removing the image. The third establishes where the underlying language model started. Holding on to all three makes it possible to distinguish an improvement associated with seeing from an improvement already present when the adapted model reads.

The paper defines multimodal gain, or MG, as image-enabled accuracy minus the same model's accuracy without images. Its multimodal leakage metric, ML, is the text-only accuracy of the adapted model minus the backbone's accuracy, with negative differences set to zero.[1]

Consider this invented example, using identical questions and scoring in every run:

Both finish at 60%. Applying MMStar's definitions gives A a gain of 20 percentage points and an ML value of 15 points; B has a gain of 35 points and an ML value of zero. These are illustrative calculations, not measurements of released systems.

The distinction changes the story told by the tie. B's result depends more strongly on supplying the image. A has acquired a larger advantage that persists without it. The table alone cannot explain how that advantage arose, but it identifies the comparison worth investigating.

Nor does B's larger gain make it universally preferable. In this example, both deliver the same accuracy when images are available. The additional measurements help explain their behavior; a user choosing a system still cares about whether it completes the actual task.

A leakage signal needs an explanation

The name ML deserves careful handling. Its formula measures a difference between model conditions; it does not inspect a training corpus. A positive result should prompt investigation of possible contamination, rather than serve as proof that particular examples were memorized. Changes in language ability or answer-format compliance are alternative explanations an evaluator would need to examine. That is a limitation of what can be inferred from the comparison.

The zero floor creates another boundary. If adaptation damages text-only performance, ML records zero. The same value also appears when the two text-only scores are equal. Keeping the raw scores therefore preserves information that the summary metric discards.

The backbone must also be the right one. A convenient language model of similar size would answer a different experimental question. When the actual precursor checkpoint cannot be tested, the three-way comparison is incomplete; the missing result should remain visible.

MMStar's submission instructions make the original backbone part of the requested evidence.[2] That requirement is especially useful when a release is accompanied by a single impressive accuracy figure. It asks the developer to expose the comparison that figure cannot supply.

The dataset is a selection, with a history

The released dataset contains 1,500 questions, divided evenly among six broad capabilities. Its card records a reduction from 22,401 initial examples to 11,607 candidates, followed by the final selection. The public rows preserve category labels and source metadata, including the earlier benchmark and split from which an example came.[3]

That provenance gives reviewers something concrete to inspect. They can identify a question's originating benchmark and split, examine its answer choices, and decide whether the image actually carries necessary information. It also makes clear that this is a curated reuse of existing material.

Selection can reduce known weaknesses without making a public test permanently immune to exposure. Once questions and answers are available, future training pipelines can encounter them. My reading is that MMStar's controls are most valuable when repeated alongside a new model evaluation, with suspicious examples inspected individually. The dataset's creation history cannot substitute for that later work.

What a current runner actually reports

The distinction between the dataset and the full experiment remains practical. EvalScope's Chinese-language MMStar documentation, inspected on November 6, 2026, describes a default validation-set evaluation with zero demonstrations, a prompt requesting step-by-step reasoning, and a final answer letter. It reports accuracy overall and by category. Its command-line example limits the run to ten questions and explicitly tells users to remove that limit for formal evaluation.[4]

Those settings describe a particular evaluation run. They do not, by themselves, supply the other two result files requested by MMStar's authors. A report headed “MMStar accuracy” can therefore be a valid result while leaving visual gain and the backbone comparison unanswered.

For a meaningful comparison, I would retain the dataset revision, model checkpoint, image processing, prompt, response parsing and sample count alongside all available scores. A change in any of them changes what the result describes. Small demonstration runs are useful for checking that the machinery works; they should keep their small denominator attached.

MMStar leaves readers with a productive habit: when a model answers a question about an image, ask what happens when the image disappears. Then ask what the model could do before it learned to accept images. The distance between those answers is where the evaluation becomes informative.

Sources

1. Lin Chen et al., “Are We on the Right Way for Evaluating Large Vision-Language Models?” — arXiv v2, April 9, 2024; affiliations and Section 4.3 metric definitions.
2. MMStar authors, official repository — the three required submission files and evaluation instructions; accessed November 6, 2026.
3. Lin Chen, official MMStar dataset release — curation counts, category balance and per-example provenance fields; accessed November 6, 2026.
4. Alibaba ModelScope, EvalScope's Chinese-language MMStar documentation — prompt, split, accuracy reporting and demonstration limits; accessed November 6, 2026.
5. Wikimedia Commons contributor 德川裕, USTC H2O Auditorium — photograph dated November 7, 2023; attribution and CC BY 4.0 license.