OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Tianbao Xie,Danyang Zhang,Jixuan Chen,Xiaochuan Li,Siheng Zhao,Ruisheng Cao,Toh Jing Hua,Zhoujun Cheng,Dongchan Shin,Fangyu Lei,Yitao Liu,Yiheng Xu,Shuyan Zhou,Silvio Savarese,Caiming Xiong,Victor Zhong,Tao Yu

2024-05-30

Abstract:Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at <a class="link-external link-https" href="https://os-world.github.io" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that the current benchmarks used to evaluate multimodal agents either lack an interactive environment or are limited to environments specific to certain applications or domains, unable to reflect the diversity and complexity of real - world computer use, thus limiting the scope of tasks and the scalability of agents. To overcome these problems, the authors introduce OSW WORLD, which is the first implemented, scalable real - computer environment that supports multiple operating systems (such as Ubuntu, Windows, and macOS) for multimodal - agent task setting, execution - based evaluation, and interactive learning. Through this environment, researchers can evaluate open - ended computer tasks involving arbitrary applications, covering aspects from image viewing to software - function integration and programming.

OSW WORLD not only provides a unified real - computer environment but also creates a benchmark consisting of 369 actual computer tasks, which involve widely - used web and desktop applications, operating - system file I/O, and workflows across multiple applications. Each task example is sourced from real - world computer - use cases and includes detailed initial - state - setting configurations and custom - made execution - based evaluation scripts to ensure the reliability and repeatability of the evaluation. Moreover, through extensive evaluations of state - of - the - art language - model (LLM) and vision - language - model (VLM) baseline agents, the paper reveals significant deficiencies in these agents as computer assistants, mainly manifested as difficulties in GUI location and operational knowledge. Through comprehensive analysis, OSW WORLD provides valuable insights for developing multimodal general - purpose agents capable of handling complex real - world scenarios.