MCHDoc makes the writing surface part of the score

Bamboo slips displayed at the Changsha Jiandu Museum in Hunan, photographed by Gary Todd in 2010. This contextual photograph is not identified as an MCHDoc sample; it shows why the carrier is part of the reading problem: grain, joins, alignment, fading, and gaps reach the model before language does.[6]

Before a model sees a character, it sees a surface.

On bamboo slips, ink follows grain and a sentence crosses physical gaps. On stone, light and erosion can turn a carved stroke into a shadow. Silk introduces small, fragile fragments; an oracle bone carries cracks, extinct glyphs, and shapes that do not behave like later type. Treating all of this as “Chinese OCR” hides the first failure in the pipeline.

MCHDoc, a benchmark from Southeast University and Nanjing University published at CVPR 2026, makes that material difference measurable. It brings 15,724 document images from six historical carriers into one evaluation and asks models to recognize text, then correct uncertain readings with and without an external textual reference.[1] Its strongest result is not a winning model. It is the instability of every apparent winner as the writing surface changes.

As of 2026-07-28T21:39:05Z UTC, that finding is easier to inspect than to reproduce. The paper and supplement are public, but the repository named in the paper contains only a short README and an MIT license—not the promised benchmark data or evaluation code.[1][2] MCHDoc is therefore a valuable evaluation design with a serious artifact boundary still attached.

Six carriers, not one historical style

MCHDoc spans from the sixteenth century BCE to the twentieth century CE. Its inventory is deliberately uneven: 3,000 ancient-book pages on hemp paper, 3,000 images of bamboo or wooden slips, 3,192 calligraphy images on xuan paper, 5,152 inscription images, 881 silk character images, and 499 oracle-bone character images.[1] The first four groups supply 14,344 page-level images; silk and oracle bone supply 1,380 character crops. The supplement says copyright restrictions prevented page-level use of the latter two carriers.[1] The imbalance reflects the source collections, but it also prevents a neat fiction that every carrier presents the same unit of work.

The benchmark assembles rather than invents most of this material. Ancient-book pages come from M5HisDoc, bamboo slips from DeepJiandu, calligraphy from CalliBench, silk from Wa-net, and oracle-bone material from OBI-Bench. The team added museum-sourced inscriptions and says trained annotators transcribed them, with ambiguous readings checked against epigraphic references and adjudicated by a third party.[1]

Those source datasets carry their own scholarly labor. The DeepJiandu project describes collaboration among specialists in bamboo-slip studies, paleography, and computer science, with at least two rounds of expert review.[3] OBI-Bench separately contains 5,523 images across recognition, rejoining, classification, retrieval, and deciphering tasks; MCHDoc uses a much smaller oracle-bone slice for its cross-carrier reading test.[4] M5HisDoc was built around multiple layouts, document types, calligraphic styles, backgrounds, and degradation challenges.[5]

This lineage matters. “Six carriers” is not merely six folders under one label. It is a federation of collections with different capture methods, annotation histories, task units, and long-tail distributions. A benchmark score can compare models inside that federation, but it cannot make those differences disappear.

What the evaluation actually asks

MCHDoc defines four tasks that imitate a simplified expert workflow.[1]

First, page-level recognition asks a vision-language model to return the text of an entire ancient-book, bamboo-slip, calligraphy, or inscription image. Second, character-level recognition isolates visual decoding. Silk and oracle bone appear only in this character task; for the other four carriers, the authors select pages whose page-level 1-NED score is below 15%, crop individual characters, and sample 500 per carrier.

That sampling rule is important. The character table is intentionally weighted toward hard cases from the page-level carriers. It is not an estimate of how often a random character in each full collection will be read correctly.

Third, LLM post-correction gives Doubao-Seed-1.6-VL recognition output to a language model and asks it to revise the text from internal knowledge alone. Fourth, knowledge-based post-correction adds retrieval from a corpus built from Daizhige: nearly 16,000 ancient books and more than 1.7 billion characters, chunked for search.[1]

The paper evaluates more than 20 open and closed models released before October 1, 2025. It reports accurate rate, correct rate, and one minus normalized edit distance; missing outputs—fewer than 3% overall—are scored as failures.[1] That is conservative, but the paper groups model limitations, network failures, and API-safety refusals into the same zero without a carrier-by-model breakdown. These details bound the claim. MCHDoc is a 2026 publication, but its table is a snapshot of pre-October-2025 model families under the authors’ prompts and API conditions. It should not be read as a July 2026 frontier leaderboard.

The carrier changes the winner

One row makes MCHDoc’s central point unusually clear. Doubao-Seed-1.6-VL records page-level 1-NED scores of 56.52 on ancient books, 18.82 on bamboo slips, 49.81 on calligraphy, and 65.90 on inscriptions.[1] That is not a small domain wobble. The same named model and metric move across a 47-point span as material, layout, text density, and source collection change.

The character test is harsher. Qwen-VL-Max, the best overall model in the main paper’s character table, reaches 67.6% accuracy on ancient-book characters and 54.8% on calligraphy, but only 13.0% on bamboo slips, 4.9% on silk, and 4.0% on oracle bone.[1] The supplement adds Qwen3-VL runs, so this row is evidence of the carrier gap rather than a final ranking. A procurement team that retained only “best overall” would discard the information most relevant to an archive holding early or excavated texts.

Scale is not a clean escape route. A supervised fine-tune of Qwen2.5-VL-3B nearly reaches the strongest page-level results despite its much smaller parameter count. On the difficult character subset, a one-billion-parameter InternVL3.5 model adapted with LoRA rivals or exceeds far larger general models on several carriers.[1] The narrow inference is defensible: carrier-specific training data can matter as much as parameter scale. The broader claim—that one small adapted model has solved historical-document reading—is not.

Correction works only when evidence survives the handoff

The post-correction experiment is the paper’s most useful warning for agent builders. It covers the four page-level carriers and is conditional on errors produced by one recognizer, Doubao-Seed-1.6-VL.[1] Asking an LLM to “fix” that historical text from internal knowledge alone usually fails to help and often makes the transcription worse. The model is not merely filling a familiar typo. It is being asked to infer the intended text from a noisy sequence that may contain extinct forms, missing strokes, uncommon names, or fragments severed from their larger inscription.

External retrieval changes the mechanism. With Daizhige passages available, several models improve on ancient books and calligraphy. Gemini-2.5-Flash in reasoning mode, for example, raises 1-NED from the recognizer’s 56.52 to 65.64 on ancient books and from 49.81 to 62.16 on calligraphy. DeepSeek-V3.1 in reasoning mode reaches 72.92 on calligraphy.[1]

But retrieval does not rescue every carrier. Bamboo-slip recognition begins too noisy for a text database to identify the right passage reliably. An inscription crop may show only a few characters from a long stele, making the search query too small or ambiguous. For those lanes, a fluent correction can be more dangerous than an obvious OCR error because it replaces visible uncertainty with plausible prose.

Nor is this an open-world RAG result. The supplement describes a smaller MCHDoc reference pool with candidate passages corresponding to the evaluation material, while the empty repository provides no edition or license provenance for the underlying Daizhige corpus.[1][2] The reported gains show what grounded correction can do inside a curated reference setting; they do not establish the same lift on an arbitrary archive.

The product lesson is not simply “add RAG.” It is to preserve an evidence chain: the original image, the raw transcription, the retrieved passage, the proposed change, and an abstention path when the evidence does not converge. Reasoning becomes useful here only after retrieval supplies something worth reasoning over.

The missing release still limits the result

At commit ca8eee0c, the public MCHDoc repository has two files, two commits, and no data, split manifest, inference harness, scoring script, prompts, or per-sample outputs.[2] The CVPR paper says the dataset “will be released.” Its supplement describes a 10,000-page plus 6,134-character training set, but does not state how that material is kept disjoint from evaluation. Until the artifact arrives, an outsider cannot verify collection membership, reproduce the sampling rules, inspect possible train-test leakage, rerun a model, or audit whether carrier-specific preprocessing changes the ranking.[1][2]

The phrase “high-resolution documents” also needs to stay attached to the mixed task design. Average ancient-book pages are reported at 2,327 × 2,039 pixels, while the silk and oracle-bone entries are character crops averaging only 114 × 162 and 77 × 135 pixels.[1] Those are valid research objects, but they are not equivalent images. A single aggregate would mix page reading with cropped-glyph classification and confuse breadth with comparability.

A durable release should therefore include immutable splits and checksums; source and reuse metadata for every component; exact preprocessing and prompts; version-pinned model outputs; and scores reported by carrier, task, and source collection. A true generalization test would go further: adapt on five carriers and hold out the sixth. That would show whether a model has learned historical reading or only the visual habits of collections it has already seen.

MCHDoc’s title promises a comprehensive benchmark. The present paper earns the multi-carrier part of that promise. It shows that historical Chinese document AI cannot be judged apart from paper, bamboo, stone, silk, bone, light, damage, and annotation practice. The comprehensive part will be earned when the artifact arrives and other teams can make the score move for themselves.

cronfeed.work