A 90-degree turn costs CFMME's top QA model 13.67 points on application questions

A Chinese business invoice photographed in Hunan in 2016. Its ruled table, faint entries, red seals, and slight skew echo the visual conditions CFMME measures; this is not a benchmark sample. Kuruman / Wikimedia Commons, CC BY 2.0; resized.[5]

A financial document may arrive as something other than a clean, upright PDF: a phone photograph taken across a desk, a table split over two pages, or a red company seal pressed through printed text. The photographed 2016 invoice above is not a CFMME sample, but it makes that intake problem tangible.[5] Before a model can reason about a tax provision or a shareholding chart, it has to decide which way is up and recover the visual structure.

CFMME, a benchmark from Alibaba Cloud's Qwen DianJin team published at ACL in July 2026, makes that ordering unusually visible. Its leading question-answering configuration, Qwen3-VL-235B-A22B-Thinking, scores 66.11 overall. On the 790-item application-question set it reaches 54.94 with the original images. Rotate every image in that set 90 degrees, without changing the questions, and the score falls to 41.27—a loss of 13.67 percentage points. At 180 and 270 degrees, it reaches 43.29 and 43.04.[1]

That result is more useful than a simple model ranking. It identifies a visual tax paid before financial knowledge can help. A second controlled test reinforces the point: when the same table questions receive lossless HTML instead of table pixels, every tested Qwen3-VL configuration improves. CFMME's main lesson is therefore not that one Chinese model is best at finance. It is that “financial reasoning” scores can conceal three different jobs: seeing the document, reconstructing it, and only then reasoning over it.[1]

The benchmark has a classroom floor and an office floor

CFMME contains 6,052 instances, but they do not form one homogeneous exam. The 2,818-item knowledge assessment draws from 43 publicly available textbooks and 20 kinds of Chinese professional qualification examinations. It includes 562 subject questions and 2,256 certification questions, all paired with images. Most use one image, while 308 use several; 560 require multiple answers, and 238 even present some answer choices as images rather than text.[1]

The 3,234 application items resemble document work. They comprise 756 seal-detection images, 875 information-extraction items, three recognition sets for seals, tables, and formulas, and 790 questions over artifacts such as candlestick charts, statistical charts, tables, and shareholding diagrams. Their sources include annual reports, research reports, invoices, websites, applications, receipts, bills, forms, and bank-card imagery. Blurring, skew, handwriting, watermarks, mobile capture, broken table rules, and seals overlapping other text are part of the test rather than noise to be cleaned away.[1]

Some material comes from earlier document benchmarks. CFMME's seal-recognition subset draws on the ICDAR 2023 ReST competition, whose original collection contains 10,000 real seal images and evaluates both locating title-text regions within those images and reading seal titles end to end.[1][3] CFMME reannotates the inherited material for complete seal-content transcription, a different output target.[1] Its table set incorporates TabRecSet, a bilingual corpus of 38,177 tables spanning scans and camera photographs, regular and distorted layouts, test papers, invoices, and other scenes “in the wild.”[1][4] CFMME is thus both a new annotation effort and a selective assembly of existing test material. Any rerun needs the identities and transformations of those inherited samples, not just the final total.

The two headline scores do not share a unit

The paper foregrounds 66.11 for question answering and 77.18 for the remaining application tasks. They look comparable because both sit on a 0-to-100 scale. They should not be read as two accuracies.

The 66.11 result belongs to the thinking version of Qwen3-VL-235B-A22B. For that headline, the paper averages its 77.27 knowledge accuracy and 54.94 application-question accuracy, which rounds to 66.11. A 77.88 weighted F1 for the full knowledge assessment—which includes items with multiple correct answers—is reported alongside those figures but is not part of the average. The aggregate therefore spans textbook knowledge, qualification questions, and visually grounded workplace reasoning, but only through the two accuracy measures.[1]

Table 4 and its caption assign the 77.18 result to the model's instruct, non-thinking configuration and say reasoning models were not run on recognition, detection, and information extraction. Section 4.2 nevertheless calls the same 77.18 result Thinking. This article follows the table because its five displayed component values are the ones being averaged, while preserving that internal label conflict.[1] The average gives equal weight to normalized edit similarity for seal text, tree-edit similarity for tables, character detection matching for formulas, mean average precision for seal localization, and field-level F1 for information extraction. These metrics reward different output structures and have different failure meanings. The average is a compact index, not the probability that a financial document was handled correctly.[1]

The evaluation conditions also resist a single horizontal ranking. All tasks are zero-shot and use one prompt within each task group, but the authors adopt each model provider's recommended temperature and sampling parameters. Open-weight models run locally with vLLM on eight Nvidia A100 GPUs; closed models arrive through APIs. Three rounds keep reported standard deviations below 1.5 points, which supports repeatability under those settings, but it does not erase differences between an API snapshot, a local checkpoint, thinking mode, and ordinary instruction mode.[1]

Rotation changes the input before it changes the answer

For the orientation experiment, the researchers rotate all 790 application-question images by 90, 180, and 270 degrees. The text of the task stays fixed. Every tested Qwen model loses more than ten percentage points in each rotated condition.[1]

The largest model is illustrative. Its 54.94 original-image score becomes 41.27 at 90 degrees, 43.29 at 180 degrees, and 43.04 at 270 degrees. The smaller 8-billion-parameter thinking model moves from 44.22 to 31.77, 31.39, and 30.89. The fact that 180 degrees is not always the worst orientation suggests that “rotation” is not one smooth difficulty dial; OCR orientation priors, page layout, and the direction of Chinese and numerical text may interact differently at each angle. That is an inference from the table, not a mechanism the paper isolates.[1]

This test has a clean deployment implication. Orientation handling is not merely cosmetic preprocessing. It is part of the evaluated system. A production document pipeline should preserve two measurements: model performance on orientation-normalized pages, which reveals downstream extraction and reasoning quality, and performance on untouched capture, which reveals the reliability of the intake path. Reporting only the first hides operational failures; reporting only the second makes a correctable capture problem look like a reasoning deficit.

The same principle applies to skew, blur, crops, and two-page tables, but CFMME publishes a paired intervention only for rotation. Its result should not be generalized into a universal robustness score for every form of document damage.

HTML exposes the cost of seeing a table

CFMME's table experiment removes a different layer of uncertainty. On a subset of application questions whose tables can be represented losslessly, the authors supply either the table image or its HTML structure alongside the same question. The large thinking model rises from 68.18 with pixels to 78.03 with HTML, a gain of 9.85 points. Its non-thinking counterpart rises from 61.36 to 75.76; the smaller thinking model rises from 65.90 to 76.51. The paper does not report how many questions are in this lossless-table subset, and both the HTML and rotation analyses cover four Qwen3-VL configurations rather than the full cross-vendor model roster.[1]

The authors describe the HTML representation as lossless: it preserves the intended table information while making rows, columns, and cell relationships explicit. That makes it an oracle structured-input comparison, not a clean estimate of perception error. Changing pixels to HTML can also change tokenization, sequence length, and how the prompt interacts with the table. The delta is therefore a representation advantage under this subset's conditions; it cannot by itself allocate errors among vision, reconstruction, arithmetic, terminology, and reasoning.

The controlled comparison nevertheless motivates a paired pipeline test. An OCR-and-layout stage may recover some of the oracle representation's advantage if it produces faithful HTML. If that stage silently moves a value into the wrong column, however, structured text may make the error look authoritative. A serious evaluation should therefore score table reconstruction separately and retain links back to the source pixels for audit.

A stamp can be legible and still be unfound

The largest metric valley appears in seal localization. CFMME asks models to return both a bounding box and a category—company seal or personal seal—for 756 document images. Of those, 528 contain both categories and 228 contain one. The best mean average precision in the table is 37.45, achieved by GLM-4.5V; Qwen3-VL-235B-A22B-Instruct reaches 30.27. Several other configurations score below one.[1]

Those detection results sit beside much stronger transcription and extraction results. The same Qwen instruct model records 88.27 in normalized edit similarity for seal-content transcription and 93.43 F1 for information extraction. Its five-metric average of 77.18 makes the weak 30.27 localization score easy to miss.[1]

This is not a contradiction. Transcribing a provided seal image and finding every seal on a cluttered page are different operations. ReST draws its boundary inside seal images: one task locates title-text regions and another reads the title end to end; its data include circular, elliptical, rectangular, and triangular seals, with curved text, background interference, and overlapping print.[3] CFMME reuses some of that imagery for full seal-content recognition and separately evaluates document-level seal boxes and categories.[1] In a financial workflow, the document-level box comes first. A model that transcribes a supplied seal image beautifully may still omit the second seal on a contract or attach the right text to the wrong object.

Model disagreement helped choose the difficult examples

CFMME did not emerge from manual annotation alone. For recognition tasks, an internal layout model first located seals, tables, and formulas. Qwen3-VL-Plus and MinerU2.5 then produced pseudo-labels; when their edit distances diverged, DianJin-OCR-R1 supplied another pass. The authors intentionally favored some high-disagreement cases because disagreement signaled difficulty, before human verification.[1]

Application questions followed a related route. Qwen3-VL-Plus generated ten candidate questions for each image. Human experts selected among them and could increase the difficulty. A human and a vision-language model then answered independently: agreement triggered a quick review by one expert, while disagreement triggered an in-depth review by three. Across the benchmark, 20 annotators contributed about 1,600 hours; specialists reviewing certification questions had passed the relevant examination.[1]

This is a practical way to spend scarce expert attention, and it creates a specific evaluation boundary. Difficulty is partly shaped by where contemporary model families disagree. That can make the benchmark excellent at finding current failure pockets while making it less neutral as a census of all financial work. A later model may improve because it solves those pockets, because its visual preprocessing resembles the annotating models, or because the task distribution no longer surprises it. Per-source and per-construction results would help distinguish those stories.

The missing artifact is the sample manifest

The ACL paper publishes prompts, model configurations, task metrics, aggregate tables, and numerous examples. Its ethics statement says the team secured permission for included datasets and that the data contain no sensitive or personally identifiable information, but it does not detail the safeguards behind that assertion.[1] The paper points readers to the official Qwen DianJin repository, whose own description presents evaluation as the first stage of an “evaluation → data synthesis → post-training” research cycle.[2]

As of September 11, 2026, the paper does not provide a public CFMME dataset manifest or a dedicated evaluation package.[1] At the inspected repository commit, the project announces the paper, but CFMME is absent from its open-source-project table and source tree.[2] That leaves outsiders unable to reproduce the exact 6,052-item mix, inspect duplicate handling, or verify image transforms and answer parsers.

Two small count discrepancies show why a manifest matters. CFMME's summary table lists 213 seal-recognition items, while the prose says 206 seal images. It lists 400 table-recognition instances, while the stated one-page and two-page breakdown is 312 plus 82, or 394. These may reflect images versus instances, post-filter additions, or simple copy errors; the paper does not reconcile them.[1] Neither discrepancy overturns the results, but both are fields a released sample ledger could settle immediately.

The strongest rerun would preserve five things:

frozen sample identifiers, upstream dataset references, licenses, and every crop or rotation applied;
exact checkpoint or API snapshot, prompt, sampling parameters, output parser, and refusal policy;
paired results for raw capture, orientation-normalized images, lossless table structure, and finance reasoning after verified transcription;
separate metrics for localization, recognition, structure, extraction, and reasoning instead of only a mixed average; and
results by source type—report, invoice, mobile photograph, website, exam, and textbook—so one abundant format cannot conceal another.

CFMME's most durable contribution is the sequence its controlled tests reveal. A financial vision model does not reason about money and then notice the page. It first has to orient the page, find the object, and rebuild the table. Turn the evidence sideways, and that first job becomes the score.

cronfeed.work