OSWorld

OSWorld is an open benchmark from the XLANG Lab at the University of Hong Kong and collaborators that tests whether multimodal AI agents can operate a real computer. Instead of answering questions or editing code, the agent sees the screen and controls the mouse and keyboard inside a virtual machine, working with ordinary desktop and web applications, files and multi-app workflows. The original paper appeared on arXiv in April 2024 with 369 tasks on Ubuntu, plus an additional 43 Windows tasks for analysis.

How scoring works. Every task comes with an initial-state configuration and a custom execution-based evaluation script that inspects the final state of the machine, for example the contents of a saved spreadsheet or a changed setting. The project lists 134 distinct evaluation functions. At release, the authors reported that humans completed more than 72% of tasks while the best model reached about 12%, with failures mostly in locating things on screen and in operating knowledge.

Versions. In July 2025 the team released OSWorld-Verified, which fixed community-reported task problems, added AWS support so a full evaluation can run in about an hour through parallelisation, and published official results run by the maintainers under consistent settings. The site says 8 Google Drive tasks can fail to initialise because of network and account issues, so results may be reported on 369 tasks or on the 361 that remain, as long as the report says which. In June 2026 the lab released OSWorld 2.0, a separate benchmark of 108 long-horizon workflows across 31 self-hosted websites and professional desktop applications. According to its paper, the median task takes a human about 1.6 hours, and agents need hundreds of steps per task, compared with about 30 in OSWorld 1.0. Its repository lists version osworld-v2.1, released in September 2026, as the recommended release.

Who it is for. Teams building or choosing computer-use agents, such as agents that fill in forms, move data between apps or test software through the interface. Researchers use it to compare models and agent frameworks.

Limits to keep in mind. Running it needs virtual machines and real infrastructure, runs are slow compared with text benchmarks, and results depend on setup choices such as screenshot resolution, step limits and which input types are allowed. Scores from OSWorld 1.0, OSWorld-Verified and OSWorld 2.0 are not interchangeable. The code is open source and free to run, and you pay only for the model and cloud resources you use. Alternatives for related skills include GAIA for browsing and tool-use questions, BrowseComp for hard web research, and Terminal-Bench for shell work. The project page is os-world.github.io and the 2.0 page is osworld-v2.xlang.ai.