Imagine asking an AI assistant to load a sales file, remove duplicate orders, and calculate revenue by region. The first two steps look uneventful. The third produces a convincing table. But suppose the assistant removed every repeated customer rather than every repeated order. The arithmetic can be flawless while the answer describes the wrong business.

That hypothetical failure is a useful way into CIBench, a code-interpreter benchmark developed by researchers at Shanghai Artificial Intelligence Laboratory and ShanghaiTech University. Its distinctive question concerns continuity: can an assistant keep solving problems when the next instruction inherits the consequences of earlier work? The paper first appeared in July 2024; this reading uses its November 6 revision.[2]

A session has a memory

The institutional context predates the paper. Shanghai AI Laboratory's January 30, 2024 announcement of OpenCompass 2.0 included CIBench among its specialized evaluations. The Chinese-language announcement described a test spanning intent recognition, programming, tool use, and reasoning through multiple conversational turns.[1] That is a more demanding promise than checking whether an assistant can supply a plausible snippet of Python.

An earlier benchmark helps clarify the change. DS-1000, introduced in November 2022, collected 1,000 data-science programming problems across seven Python libraries, drawing on Stack Overflow. Its evaluators check functional behavior and, where required, restrictions on how a solution is written.[3] These are valuable tests: a program can return the expected value while violating an instruction to use a particular operation.

CIBench adds connected questions in interactive IPython sessions.[2] The difference becomes tangible in the sales example. A standalone deduplication problem can give the model a clean table and ask it to remove the appropriate rows. A continuing session must also carry forward the table it actually created. The next request arrives in a world partly made by the assistant.

The released prompt makes this explicit. It tells the agent that execution results persist, requires interpreter use, and asks for code limited to the current step. The supplied configuration wraps the model in a ReAct-style agent with an IPython interpreter and sets max_turn=3.[4] This is a specified interaction budget, not an invitation to keep trying indefinitely until something works.

Two histories, two questions

CIBench's end-to-end mode continues from the model's own previous work. Its oracle mode supplies reference code and reasoning as context for subsequent questions, approximating assistance.[2] The word oracle describes an evaluation condition; it does not mean the assistant has independently become more reliable.

Return to the hypothetical sales analysis. One run reaches the regional-revenue question after deleting the wrong rows. Another reaches it with the correct preparation available in its history. A better answer in the second run would be useful evidence: the assistant may be able to perform the later operation when the earlier context is right. It would leave a different question unresolved: can the assistant notice and repair its own mistaken preparation?

The distinction also prevents an unfair diagnosis. If a model's last answer is wrong, that alone does not establish that it cannot perform the last operation. It might be calculating correctly on damaged inputs. Conversely, success after receiving a reference history cannot establish that it would have produced that history itself.

My reading is that these modes make the location of assistance visible. They separate two plausible products: an analyst's supervised helper and an assistant entrusted with a complete sequence. Both can be useful. Their scores answer different questions, so a report should name the mode beside the result.

Running code is only the first inspection

The paper separates tool invocation and error-free execution from output assessment. It evaluates numerical answers, structured text using ROUGE, and visual output using structural similarity to reference images.[2] Each measurement illuminates a different part of the job.

In the sales example, successful execution would mean the interpreter accepted the code. It would not mean the assistant chose the right definition of a duplicate. Even an attractive regional plot could conceal that decision. This is why an execution score should not quietly become a claim about analytical correctness.

The output measures also need interpretation. Similar wording or a similar-looking plot is evidence of resemblance under the chosen metric. A reader still needs to know whether the requested quantity, grouping, and transformation were preserved. For a practical evaluation, I would inspect a sample of complete transcripts alongside the scores, following the data from its first load to the final output.

CIBench's authors acknowledge limits involving Python-only coverage and the evaluation of tasks with randomness, including model training.[2] Those boundaries matter when carrying a benchmark result into a new workflow. A reproducible table transformation and a stochastic training run do not offer the evaluator identical notions of a correct output.

What should travel with the score

The repository's setup instructions require OpenCompass, a downloaded CIBench dataset, and a separate script to collect its underlying data sources. They also distinguish saved evaluation results from output images.[4] The benchmark therefore has several inspectable artifacts beyond the final average.

For anyone comparing Chinese model families on this task, the useful evidence package would identify the checkpoint or dated API version, dataset split and language, agent prompt, interpreter environment, interaction budget, and evaluation mode. Hardware and serving configuration belong beside any latency or throughput claim. Those are reporting requirements I would apply to a new comparison, rather than results this article claims to have measured.

This is a reading of a 2024 evaluation design, not a fresh ranking of 2026 models. Its durable lesson is easy to carry into the next model release: when an assistant finishes an analysis, ask what happened to the notebook along the way. A final answer becomes much more informative when the record shows who supplied the correct history.

Sources

1. Shanghai Artificial Intelligence Laboratory, “OpenCompass 2.0 evaluation system released” (January 30, 2024; Chinese first-hand announcement describing CIBench's multi-turn evaluation scope).
2. Chuyu Zhang and colleagues, “CIBench: Evaluating Your LLMs with a Code Interpreter Plugin,” arXiv version 3 (November 6, 2024; affiliations, connected sessions, evaluation modes, metrics, and limitations).
3. Yuhang Lai and colleagues, “DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation” (November 18, 2022; problem construction and functional and surface-form evaluation).
4. OpenCompass team, CIBench official repository (README setup instructions and the released cibench_eval/prompts.py and eval_cibench_hf.py files; inspected October 26, 2026).
5. ShanghaiTech University, “Campus aerial photography, August 2022” (published October 18, 2022; official gallery and source of the article photograph).