Hanfu-Bench asks AI to modernize a garment without erasing its dynasty

A manufacturer-sponsored Hanfu show at Beijing’s 32nd IDO ACG Expo on January 18, 2020. The photograph is not part of Hanfu-Bench; it shows the contemporary styling boundary that the benchmark tries to formalize. N509FZ / Wikimedia Commons, CC BY-SA 4.0; resized.[5]

Modernizing a historical garment sounds easy until the success condition is stated precisely. The new design must change enough to belong in contemporary life, yet preserve enough of the original construction to remain culturally legible. A wide sleeve is not merely an atmosphere. A collar, skirt, lapel, layer, and period attribution form a grammar; alter the wrong feature and “adaptation” becomes substitution.

Hanfu-Bench, published at EMNLP in November 2025, turns that tension into two linked tests. First, can a vision-language model identify period-sensitive features in photographs of Hanfu? Second, can a caption-and-image pipeline carry those features into modern clothing? The benchmark’s most useful result is not that one model leads a table. It is that recognition and reinvention fail differently—and that several apparently small evaluation choices can change what the score means.[1]

The wardrobe is contemporary, even when the label is historical

Hanfu-Bench began with 902 outfits. Its authors collected product photographs from seven Hanfu retailers, screenshots from seven well-reviewed Chinese historical television series, and photographs contributed by members of a university Hanfu club. After duplicate removal, quality filtering, cropping, and annotation, the released set retained 496 outfits across 1,192 images. Nine annotators with Hanfu knowledge—four paper authors and five club members—worked in groups of three; an image survived only when all three approved its clarity and the visibility of its features.[1][2]

That provenance defines the benchmark. These are not 496 excavated garments, museum accessions, or securely dated textiles. They are contemporary representations of historical dress: retail products, screen costumes, and clothes worn by enthusiasts. The benchmark therefore measures whether a model can read the visual taxonomy used in today’s Hanfu ecosystem. It does not, by itself, establish archaeological accuracy.

The taxonomy is still unusually concrete. Annotators recorded three global attributes—garment type, intended gender, and period—and five local ones: sleeve, collar, jin or front/lapel construction, bottoms, and outerwear. “Type” separates traditional form from improved Han clothing and modern Han-element clothing. Period labels apply to outfits classified as traditional. If three annotators could not agree on a feature, the value became “unsure,” rather than being forced into a clean category.[1][2]

From those labels, the authors generated 1,721 single-image questions and 2,465 multi-image questions. Each multiple-choice item was constructed to contain exactly one valid answer. The difference matters: a single-image item asks what one garment is; a multi-image item can ask which candidate comes after the Tang dynasty or which shares a particular structural feature. One is recognition. The other requires comparison.[1]

An 82.28 score contains a great deal of gender recognition

On the full question set, Doubao-1.5-Vision led the five tested models with 82.28% on single-image VQA and 75.95% on multi-image VQA. GPT-4o followed at 77.69% and 69.53%. The three open-weight systems—MiniCPM-V 2.6, Qwen2.5-VL-7B-Instruct, and InternVL2.5—were more uneven, especially when several images had to be compared.[1]

Those overall figures should not be read as a pure test of historical knowledge. Gender was both common in the question mix and easy for the systems: the five models scored roughly 88% to 96% on single-image gender items and about 59% to 95% on the multi-image version. Period recognition was much less settled. Single-image period accuracy ranged from 42.75% to 76.81%; the spread itself is more informative than an overall average that mixes period, sleeve, gender, and outerwear.[1]

The paper’s main table reports, for each model, its best result among five Chinese role prompts rather than fixing one neutral instruction across the leaderboard. The headline scores are therefore prompt-selected upper envelopes, not one like-for-like prompt run. Elsewhere, the authors show that single-image results move more with prompt wording. Telling the model to assume the role of a traditional-clothing expert could even lower performance on some attributes. Asking for step-by-step reasoning or a rationale produced no consistent gain.[1] A deployer should therefore preserve the exact Chinese prompt, model snapshot, image preprocessing, answer parser, and per-attribute slices before treating a rerun as comparable.

The model version matters as much as the prompt. Volcengine’s official January 2025 announcement described Doubao-1.5-Vision-Pro as an upgrade in dynamic resolution, multimodal alignment, fine-detail understanding, and instruction following.[4] Hanfu-Bench tested that generation of systems. Its ranking is a dated baseline, not a verdict on whatever a provider calls “Doubao,” “Qwen,” or “InternVL” in September 2026.

Comparison helps people in a way the aggregate table obscures

For the human comparison, the researchers created balanced subsets of 160 single-image and 160 multi-image questions—20 for each of the eight attributes. Three Hanfu-familiar evaluators and three non-experts answered them. Experts rose from 77.50% on single images to 83.54% when they could compare several images; non-experts rose from 57.92% to 64.58%. Interviewed evaluators said that comparison let them isolate differences and learn from earlier questions.[1]

The models did not share one pattern. Doubao also improved on the balanced subset, from 70.00% to 74.48%, and GPT-4o stayed almost flat. All three open-weight models declined, with MiniCPM-V 2.6 falling from 52.50% to 27.50%. This is why “multi-image is harder” is only a directional summary. It is true for every model in the larger, differently composed table, but not for every model once the human-comparison subset balances attributes.[1]

The human baseline is small—six people, split into two groups—and should not be generalized to “human ability.” Its better use is diagnostic. Humans could turn neighboring examples into a temporary visual lesson. Most tested open models treated the extra images more like additional input load than like a comparison set. A stronger follow-up would test the same outfits under controlled contrast: one changed collar, one changed sleeve, one changed period, with difficulty and distractor distance reported explicitly.

The background may be context, or it may be a shortcut

The core dataset removes irrelevant background details so that clothing remains the target. In a supplementary condition, the authors restored scene context. All five models improved overall, particularly on global attributes and gender, although some local-feature judgments became noisier. Chinese instructions were also more consistent than English ones on single-image questions.[1]

Both findings are useful, but neither has a single interpretation. A television set, hairstyle, prop, or retail staging can supply legitimate period context. It can also let a model guess “Tang” from scenery without reading the garment. Likewise, an English prompt may lose distinctions carried compactly by Chinese clothing terms, or it may simply differ in phrasing quality. To separate knowledge from shortcut, the next evaluation needs source-stratified results—retailer, television, and club photography—and counterfactual pairs in which the same outfit is placed against different backgrounds.

That source split is especially important because the released dataset card exposes the images and labels for research evaluation but restricts the collection to academic, non-commercial, evaluation-only use.[2] The public code repository makes the single- and multi-image runners and the transcreation stages inspectable.[3] Reproducibility is therefore better than a closed leaderboard, but the dataset is designed as a test set, not a training recipe for correcting the failures it reveals.

Forty-two percent belongs to a pipeline, not to one model

The second task is the more original one. The researchers selected 50 visually distinctive traditional outfits. GPT-4o first described each source image, then rewrote the description as a modern fashion concept. During that edit, Chinese captions were translated into English for compatibility. Three image systems—InstructPix2Pix, Stable Diffusion 2.1, and Stable Diffusion XL 1.0—then produced 150 outputs. Five Hanfu experts, blinded to which generator made which image, scored visual change, semantic equivalence, naturalness, modern adaptability, cultural inheritance, and attractiveness on five-point scales.[1][3]

An output counted as successful only if it scored at least three on semantic equivalence, modern adaptability, and cultural inheritance. Stable Diffusion reached the highest success rate, 42%; InstructPix2Pix reached 8%. Stable Diffusion XL scored highly on inheritance partly because it changed the input less. That is not a footnote—it is the central tradeoff. A system can preserve tradition by refusing to modernize, or modernize by discarding the construction that made the source specific.[1]

The paper’s case study shows how the cascade can drift. A Tang outfit with straight sleeves is misdescribed as having wide, flowing cuffs; the modernized prompt faithfully carries the wrong feature forward. The final image may look coherent while inheriting an error introduced before generation. Because GPT-4o performs visual description and caption editing, translation intervenes, and a separate diffusion model renders the result, the 42% rate cannot be assigned to a single “Hanfu model.” It measures an entire 2025-era chain.[1]

That boundary becomes more important with time. None of the three tested image generators represents the September 2026 frontier, and the pipeline does not test a current unified model that can inspect, revise, and explain its own output. A fair modern rerun should keep the 50 source images and expert rubric, then report failure at each handoff: feature extraction, Chinese-to-English terminology, modernization plan, and rendering. Otherwise a beautiful output can conceal a lost collar, and a faithful output can conceal that almost nothing changed.

Hanfu-Bench changes the evaluation question in a productive way. Cultural vision is not only whether a model can name an object. It is whether the system can identify which features carry history, distinguish those features from contemporary staging, and transform the rest without laundering a mistake into style. The benchmark does not finish that measurement. It gives it a wardrobe, a timeline, and a failure condition.

cronfeed.work