Imagine handing an assistant a thick project history and asking why a decision was reversed. Finding the original decision is useful. Connecting it to a later objection and the final amendment is the work. A system can have enough room for the entire file and still fail that second task.

LongBench v2, developed by researchers at Tsinghua University and Zhipu.AI, makes this distinction concrete.[1] Released on December 20, 2024, it remains worth reading as an evaluation design.[4] The historical results below belong to its original experiments; they are not a ranking of models available in September 2026.

A different kind of reading exam

The first LongBench, published at ACL in August 2024, assembled 21 datasets across six task categories in English and Chinese. It included summarization, question answering, few-shot learning, synthetic tasks, and code completion. Its average English input was 6,711 words; the Chinese average was 13,386 characters.[2]

That breadth matters when comparing generations. A benchmark containing summaries and code completions measures several kinds of output. Changing the task format changes what a score can tell us, even before documents get longer. A percentage from one version cannot simply be placed beside a percentage from another as if they were successive school grades.

The v2 dataset contains 503 four-option questions, with contexts ranging from 8,000 to two million words. Its six categories cover individual documents, document collections, learning from examples, conversation histories, code repositories, and structured data. The official dataset labels the language English and provides fields for domain, subdomain, difficulty, and length.[3]

Those fields invite a more useful reading than a single average. A repository question may depend on relationships among functions; a conversation question may depend on which instruction superseded another. That is my interpretation of why task-level results matter: success in one kind of reading need not transfer evenly to another.

The language boundary also deserves attention. Research produced in China's AI ecosystem can supply a global English-language test. Its institutional origin does not make its aggregate score evidence of Chinese-language comprehension. The earlier bilingual benchmark and this English dataset support different conclusions.[2][3]

The missing middle

The public evaluation code exposes a consequential detail: when a prompt exceeds the configured token limit, it keeps the beginning and end and removes the middle. The dataset's word count and the model's actual input are therefore different quantities. Tokens are tokenizer units, not words.[4]

Return to the hypothetical project history. Suppose the first pages approve a design, the middle records an exception, and the final pages report the outcome without repeating the reason. Remove the middle, and the assistant loses the premise needed to explain the reversal. More time thinking about the surviving pages cannot reconstruct that missing evidence reliably.

This suggests a practical distinction for any document evaluation: separate questions whose decisive evidence reached the model from questions whose evidence was discarded. Both outcomes matter for a deployed system, but they diagnose different problems. The first group can expose failures of interpretation. The second can expose failures in how information is selected before interpretation begins. This is a proposed diagnostic, not a reported LongBench v2 result.

The same repository makes another choice inspectable. Its ordinary chain-of-thought path allows up to 1,024 output tokens, then asks for a final answer in a separate call capped at 128 tokens. That second prompt contains the earlier reasoning and answer choices while omitting the document.[4] My reading is that the intermediate response becomes a compressed working record: a detail left out of it is unavailable to the final call.

What the famous percentages establish

In the paper's ordinary-accuracy table, GPT-4o-2024-08-06 scores 50.1% with direct zero-shot prompting. o1-preview-2024-09-12 scores 57.7%, using its built-in reasoning. The human baseline is 53.7%, with document-search tools and a 15-minute limit per question; participants abstained on 8% of questions.[1]

The four-percentage-point lead is real within that comparison. It establishes performance against a particular timed human procedure. It does not establish superior reading under every working condition. Nor is comparing two different models a controlled experiment that isolates the effect of extra reasoning time. These are reported accuracy results, not measurements of latency, hardware efficiency, or cost.[1]

For a useful follow-up, I would hold the model and delivered evidence fixed, vary its reasoning budget, and retain the full responses. Then I would repeat the comparison on questions where all necessary evidence fits. That would make it easier to tell whether extra computation helps combine available facts, rather than allowing changes in model identity or input coverage to explain the difference.

What to carry into the next evaluation

LongBench's authors made their evaluation choices inspectable through code, prompts, data, and documentation, including a Chinese-language account of the original benchmark.[4] That is a valuable contribution to the evidence behind long-context claims.

When assessing a document assistant, I would preserve the actual input after truncation, the intermediate response, and the final answer alongside the score. I would also ask for an explanation tied to the decisive passages on a small, manually reviewed sample. A correct letter alone cannot show whether the assistant followed the relationship that mattered.

The project-history question gives this work a clear purpose. The desired assistant can recover the objection, connect it to the amendment, and explain the reversal. A larger document allowance creates room for that evidence. Evaluation has to establish what happens after it arrives.

Sources

1. Yushi Bai et al., “LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks,” arXiv revision 2 — affiliations, original ordinary-accuracy results, model versions, and human evaluation protocol.
2. Yushi Bai et al., “LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding,” ACL, August 2024 — original benchmark scope, languages, tasks, and average input lengths.
3. Z.ai / LongBench team, official LongBench-v2 dataset — question count, language, context lengths, task categories, and record schema; accessed September 22, 2026.
4. THUDM, LongBench repository, commit 2e00731 — release history, pred.py, prompts/0shotcotans.txt, and LongBench/README_ZH.md; public inference procedure and Chinese-language firsthand documentation.
5. Lining Huo, “Night View of the West Library in Tsinghua University,” November 30, 2022 — original photograph and provenance, Wikimedia Commons, CC BY-SA 4.0.