WebArena: A Realistic Web Environment for Building Autonomous Agents

Shuyan Zhou,Frank F. Xu,Hao Zhu,Xuhui Zhou,Robert Lo,Abishek Sridhar,Xianyi Cheng,Tianyue Ou,Yonatan Bisk,Daniel Fried,Uri Alon,Graham Neubig

2024-04-16

Abstract:With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands. However, current agents are primarily created and tested in simplified synthetic environments, leading to a disconnect with real-world scenarios. In this paper, we build an environment for language-guided agents that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management. Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage human-like task-solving. Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions. The tasks in our benchmark are diverse, long-horizon, and designed to emulate tasks that humans routinely perform on the internet. We experiment with several baseline agents, integrating recent techniques such as reasoning before acting. The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%. These results highlight the need for further development of robust agents, that current state-of-the-art large language models are far from perfect performance in these real-life tasks, and that WebArena can be used to measure such progress.

Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that current autonomous agents are created and tested in simplified artificial environments, which leads to a disconnection from real - world scenarios. Specifically, the author points out that existing evaluation environments often oversimplify real - world situations. This not only limits the diversity of tasks but also reduces the complexity of tasks and fails to comprehensively reflect the challenges in the real world. Moreover, these environments are usually presented as static resources, limiting the breadth and diversity of agent exploration.

To address these issues, the paper proposes **WebArena**, a highly realistic and reproducible web environment aimed at promoting the development of autonomous agents capable of performing tasks. WebArena contains four fully functional self - hosted web applications, each representing a common area on the Internet: online shopping, discussion forums, collaborative development, and business content management. In addition, WebArena also integrates multiple utility tools (such as maps, calculators, and notepads) as well as knowledge resources (such as the English Wikipedia and domain - specific reference manuals) to support human - like task execution.

The paper evaluates the functional correctness of these tasks by introducing a benchmark test suite containing 812 long - term web tasks. These tasks are designed to mimic the tasks that humans perform daily on the Internet, covering three major categories: information search, site navigation, and content and configuration operations. The experimental results show that even the most advanced large - language models (such as GPT - 4) are far from perfect in these practical tasks, with an end - to - end task success rate of only 14.41%, while humans perform as high as 78.24%. This result emphasizes the need for further development of more powerful and effective agents, especially in terms of the ability to handle complex tasks, such as active exploration and failure recovery.

In conclusion, through the construction of WebArena and its accompanying benchmark test suite, this paper provides a new platform for evaluating and promoting the development of autonomous agent technology, with a particular focus on improving the performance of agents in complex real - world tasks.