OSWorld

OSWorld is a benchmark for researchers and developers testing AI agents that use computers. It measures task completion in real desktop environments, including workflows that cross application boundaries.

• Desktop and web tasks: Evaluate agents on Chrome, GIMP, LibreOffice, VS Code and file operations, with defined starting states and checks of the resulting work.
• Custom evaluations: Connect your own agent, add tasks involving other applications, or use the environment for interactive learning.
• Parallel runs: Run multiple desktop environments on one host, with headless operation available for larger evaluations.
• Task inspection: Manually examine tasks and evaluation checks, and record actions through screenshots and videos.

The environment supports Ubuntu, Windows and macOS. Local execution uses VMware or VirtualBox, while Docker supports server deployments with KVM recommended for performance. AWS, Modal and Daytona offer remote execution alternatives. The project is open source under Apache 2.0.

Agent baselines include UI-TARS, Qwen, GPT and Claude; the GPT baseline can use a custom OpenAI-compatible API endpoint. Local desktop execution doesn't make every evaluation offline: cloud model APIs receive requests outside your machine, and some tasks depend on network access. Google account tasks require account access, and Google Drive tasks can be excluded when network dependencies prevent them from running.