An AI assistant spends hundreds of actions assembling an overseas reimbursement claim. It reads instructions, searches transactions, and gathers evidence. When it finally reaches the expense portal, most of its allowance is gone. At step 500, it is still working out how to enter a date. The claim remains unsubmitted. This is a recorded failure case in OSWorld 2.0, and it captures an office frustration that a fluent answer can hide: preparation has consumed the time available for delivery.[1]

The benchmark comes from a collaboration led by XLANG Lab at the University of Hong Kong. Its project page describes 108 extended computer workflows, with a median human completion time of about 1.6 hours. These are assignments large enough for an assistant to lose track of what it owes the user.[1]

As of October 19, 2026, the important distinction is between the research evaluation and the software used to reproduce it. OSWorld 2.0 introduced the longer workflows in June; the repository announced the osworld-v2.1 maintenance release on September 16. Reading their results requires keeping both the meaning of success and the benchmark version attached.[3][5]

From operating an application to carrying an assignment

The original OSWorld paper, submitted in April 2024, established a useful baseline: 369 tasks involving real desktop and web applications, files, and work spanning multiple programs. Each task came with an initial computer state and an execution-based evaluator. An agent had to change something in the environment that could be checked.[2]

That design matters because a description of a successful action is easy to confuse with the action itself. Explaining how to edit a spreadsheet leaves the spreadsheet untouched. Testing the resulting state connects an answer to its consequence. The original benchmark made that connection explicit, giving researchers a way to examine where agents struggled with interfaces and software operations.[2]

XLANG’s broader research agenda helps explain the continuity. The lab describes its purpose as grounding language and perception in executable actions across databases, computers, and physical systems. It announced OSWorld 2.0 on June 26, 2026.[3] For this part of China’s AI research ecosystem, the contribution includes the measuring apparatus: defining a task carefully enough that different agents can attempt it and researchers can inspect what happened.

Two scores answer different questions

OSWorld 2.0 scores the final environment against task-specific checkpoints, allowing different valid routes to a result. Partial credit records how much of the rubric is satisfied; binary completion requires all scoring checkpoints to be fully satisfied. The paper reports an average of 27.25 checkpoints per task.[4]

In its Table 3, Claude Opus 4.8 with maximum thinking and batched tool calls achieves 54.8% partial credit but 20.6% binary completion. These are reported results across the 108 tasks using screenshot observations, a 500-step budget, and release v2026.06.24. The experiments ran headlessly on AWS CPU instances; batched actions can include several tool calls within one step.[4]

The gap is informative. Partial credit distinguishes substantial progress from an empty attempt. Completion asks whether the assignment meets the entire scoring rubric. Neither percentage directly measures hours of human work saved. A reviewer might repair one missing field quickly, while an apparently extensive result could require checking every row. Turning these scores into productivity claims would require an additional study of that repair and review work.

The test environment belongs beside the score

A result also depends on which world the agent encountered. The repository describes September’s 2.1 release as a bug-fix update with revised agents, evaluation workflows, and pinned component references. Its setup instructions require matching versions of the code, task files, assets, and simulated websites. The official task implementations are distributed through gated datasets to reduce leakage of answers and evaluator details into an agent’s online searches.[5]

This is a practical boundary on leaderboard reading. A changed task or evaluator can change a result even when the model has not changed. The June paper’s numbers therefore should not be relabeled as fresh 2.1 measurements. A newer version can improve the test while requiring a new comparison.

The project’s release documentation makes this unusually concrete. A manifest identifies the task and asset versions, website code, evaluation code, and virtual-machine images intended to belong together. It also distinguishes those shared inputs from the provenance of an individual run. Different infrastructure providers count as comparable only when the maintainers have verified that their images are sufficiently equivalent.[6]

In other words, naming a model and writing “OSWorld” beside its score leaves essential information out. The release manifest documents the shared starting conditions; the individual run still needs its own record.[6]

What a more persuasive result would show

My reading is that this benchmark encourages two kinds of progress to be reported together: improvements in finishing assignments, and evidence that the comparison itself remained stable. A future result would be more useful with its release manifest, task-level outcomes, and examples of remaining failures than with a single percentage detached from its conditions.

The reimbursement case supplies a good question to carry into those examples: did the assistant leave the user with a submitted claim, or with more preparation to finish? A research score can recognize the preparation. The person waiting for reimbursement still needs the claim to go through.

Sources

1. OSWorld 2.0 project team, “OSWorld 2.0: Benchmarking Computer-Use Agents on Long-Horizon Real-World Tasks”; affiliations, task statistics, and Task008 reimbursement failure trajectory, accessed October 19, 2026.
2. Tianbao Xie et al., “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments,” first submitted April 11, 2024; original tasks and execution-based evaluation.
3. XLANG Lab, University of Hong Kong, official research overview and June 26, 2026 OSWorld 2.0 announcement.
4. Mengqi Yuan et al., “OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks,” arXiv version 1, 2026; Sections 2.1.3 and 3.1–3.2, Table 3, and Appendix G.3.1 on scoring and evaluation conditions.
5. XLANG Lab, OSWorld-V2 repository README; September 16, 2026 release announcement, matching setup components, and gated task distribution, accessed October 19, 2026.
6. XLANG Lab, “Benchmark Releases,” OSWorld-V2 documentation; release manifests, individual-run provenance, and provider comparability, accessed October 19, 2026.
7. Adon3465, “University of Hong Kong Main Building,” photograph dated January 30, 2013, Wikimedia Commons; image provenance and CC BY-SA 3.0 license.