TM-Bench gives Traditional Mongolian a scorecard of its own

Academician D. Tömörtogoo examines a Mongolian manuscript in a photograph published by the National Library of Mongolia. The image shows the material and expert tradition that a machine-readable benchmark tries to preserve without flattening it.[6]

“Multilingual” is one of AI's most elastic adjectives. A model can earn it by performing well across a bundle of high-resource languages while an entire writing system remains nearly invisible inside the average. Traditional Mongolian is a sharp test of that invisibility: a model may render its characters, recognize a few familiar patterns, and still fail to produce a coherent sentence.

TM-Bench, created by researchers at Inner Mongolia University and accepted as a resource paper at SIGIR 2026, gives that failure a dedicated place to appear. Its 18,357 examples cover five kinds of language work across understanding and generation. The authors report that most tested models fell sharply on the understanding tasks, with only a few slightly above the relevant random baseline, while generated Traditional Mongolian often deteriorated into incoherent or unreadable text.[1][3]

That result is more useful as a diagnosis than as a leaderboard. TM-Bench shows where broad multilingual claims stop carrying information, but its construction mix, one-direction generation test, and still-incomplete public evaluation harness also set limits on what can be concluded from it.

The blind spot inside “multilingual”

Traditional Mongolian is not simply another line in a translation spreadsheet. The script runs from top to bottom, with columns progressing left to right. Its letters join cursively, and suffixes can be separated from a word by a narrow no-break space that still belongs to the word and affects shaping. The W3C's layout requirements devote separate rules to selection, punctuation, line breaking, controls, and mixed horizontal–vertical content.[5]

TM-Bench does not evaluate browser typography. It works with text stored in JSON. Yet the writing system's digital mechanics matter to the evaluation contract. A character sequence can be visually malformed, normalized differently, or split at the wrong boundary before a model's semantic ability is even tested. The repository acknowledges this layer with a correction utility for Mongolian character encoding and spacing.[2] A durable leaderboard therefore needs to preserve code points, normalization rules, font and renderer assumptions, as well as the prompt and answer.

This is the first important distinction in the benchmark: machine-readable is not the same as linguistically usable. A test can parse cleanly while the displayed text is wrong; a model can output valid Unicode while the sentence is nonsense. Those are different failures and should remain separately observable.

Nine files, five kinds of work

The release contains nine datasets. Eight measure natural-language understanding through topic classification, semantic similarity, natural-language inference, and multiple-choice question answering. The ninth, TM-CMMT, contains 1,995 Chinese-to-Traditional-Mongolian translation examples. Together the files total the advertised 18,357 records.[2]

Their construction is deliberately hybrid. Depending on the file, 48–60% of examples come from translation-based adaptation, 10–20% from expert-original authoring, and 30–37% from semi-automated synthesis. The culturally specific portions introduce grassland life, folklore, history, policy, ethnic geography, local textbooks, traditional medicine, dialogue, and literature instead of relying entirely on translated generic benchmarks. The team reports multiple rounds of review by native expert linguists and a Cohen's κ of 0.81 for annotation agreement.[1][2]

The mixture solves one problem and creates the next audit question. Adaptation supplies scale and cross-language comparability; expert writing supplies local idiom and knowledge; synthesis expands coverage. But a model can perform differently on each provenance lane. A single aggregate score may conceal whether it learned Traditional Mongolian, exploited the recognizable structure of an imported benchmark, or benefited from artifacts introduced during translation. Future result tables should therefore report performance separately for translated, expert-authored, and synthesized items, not only by task.

Recognition and writing fail differently

TM-Bench's understanding side asks bounded questions: choose a topic, judge whether two passages are equivalent, infer a relation between a premise and hypothesis, or select one answer from a short list. “Slightly above random” has no universal value here because chance differs across binary, three-way, and four-way tasks. The relevant comparison belongs to each task's label space, prompt format, and few-shot setup.[2]

Even so, the reported direction is stark. Among representative Llama, Qwen, and Gemma models, only a small number—including Gemma 3 27B and Qwen3 32B in the authors' summary—rose modestly above chance on the understanding suite. That is evidence of a large capability gap under this setup, not proof that those models possess no Traditional Mongolian competence anywhere else.[1][2][3]

The generation result is more visceral. On TM-CMMT, the authors used automatic metrics and double-blind human evaluation and report “semantic collapse”: outputs lost coherence and sometimes became unreadable strings.[1][3] Classification can occasionally survive on lexical fragments or option structure; translation has to sustain syntax and meaning across a full sequence. Keeping those outcomes separate prevents a tolerable multiple-choice score from being mistaken for writing ability.

The boundary is equally important. TM-CMMT is one generation dataset, one tested direction, and 1,995 examples. It does not establish the quality of every conversational, literary, summarization, or reverse-translation use case. It establishes that Chinese-to-Traditional-Mongolian generation was a serious weakness for the tested model snapshots under the paper's evaluation.

What outsiders can verify today

The public artifact makes the corpus tangible. At the repository snapshot inspected on July 20, 2026, an outsider can count all nine JSON files, inspect their schemas and labels, read the few-shot prompts, and see the character-correction code. This is enough to audit the task inventory and reproduce the headline sample count.[2]

It is not yet enough to reproduce the published comparison end to end. Although the README promises complete evaluation code and baseline result files, the tracked tree at commit 027df4f contains neither those result files nor a full model-running and scoring harness. The correction utility also refers to a Java component that is not included in the repository snapshot.[2] Those omissions do not invalidate the paper's finding; they prevent an independent team from checking exactly how much of the gap comes from model capability, prompting, decoding, normalization, rendering, or scoring.

A strong next release would pin the evaluated model revisions, inference parameters, prompts, fonts and normalization pipeline; publish per-item outputs and native-review rubrics; and provide a containerized scorer. It should also expose provenance-stratified results and confidence intervals. Those details would turn TM-Bench from a valuable first measurement into a comparison that can survive new models and new text stacks.

A local evaluation institution comes into view

TM-Bench also broadens the map of China AI. Inner Mongolia University's National–Local Joint Engineering Research Center for Mongolian Intelligent Information Processing was approved in 2019. Its stated agenda includes Mongolian speech, language-data software, cloud services, and applied information processing.[4] A benchmark built by that ecosystem carries knowledge that a general model lab is unlikely to reconstruct from translated web text alone: which distinctions matter, where encoding breaks, which cultural domains deserve tests, and when output is readable to an expert.

That may be TM-Bench's most durable contribution. It supplies a place where failure in Traditional Mongolian cannot be averaged away. The current release still needs a complete reproducibility path, and its generation evidence should stay attached to the tested direction. But the scorecard now exists, the data can be inspected, and native expertise sits inside the benchmark rather than outside it as an afterthought.

cronfeed.work