AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

Christopher Rawles,Sarah Clinckemaillie,Yifan Chang,Jonathan Waltz,Gabrielle Lau,Marybeth Fair,Alice Li,William Bishop,Wei Li,Folawiyo Campbell-Ajala,Daniel Toyama,Robert Berry,Divya Tyamagundlu,Timothy Lillicrap,Oriana Riva

2024-10-23

Abstract:Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device's system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at <a class="link-external link-http" href="http://github.com/google-research/android_world" rel="external noopener nofollow">this http URL</a>.

Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problem this paper attempts to address is the lack of realistic and repeatable benchmark environments for evaluating autonomous agents in mobile settings. Specifically, existing evaluation methods mostly rely on static test sets, which fail to adequately reflect the variations in task parameters and conditions in the real world, leading to potentially inaccurate assessments of agent performance. Therefore, the paper proposes a new environment called ANDROID WORLD, which aims to address these issues by dynamically generating tasks, providing real-world diversity and scale, and ensuring the repeatability of each task.

### Main Contributions:
1. **Creation of a new, highly diverse, and realistic mobile UI control agent environment**: ANDROID WORLD covers 20 Android applications with 116 programming tasks, providing real reward signals based on device states.
2. **Establishment of benchmark performance**: Demonstrated the performance on ANDROID WORLD using a state-of-the-art multimodal agent M3A, achieving a success rate of 30.6%.
3. **Detailed analysis**: Showcased the importance of evaluating agents under different task parameters and conditions, highlighting the impact of inherent randomness in models and environments on performance.

### Problems Addressed:
- **Lack of realistic evaluation environments**: Existing environments are mostly focused on desktop computing, ignoring the diversity and complexity of mobile devices.
- **Limitations of static test sets**: Existing environments only support static task parameters, unable to cope with the wide variations in task inputs in the real world.
- **Reliability of reward mechanisms**: Current methods rely on human or language model judgments to determine task completion, which are unreliable or difficult to scale in large-scale applications.

Through these contributions, ANDROID WORLD provides a more comprehensive, realistic, and reliable benchmark environment for evaluating mobile autonomous agents.