AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Christopher Rawles,Sarah Clinckemaillie,Yifan Chang,Jonathan Waltz,Gabrielle Lau,Marybeth Fair,Alice Li,William Bishop,Wei Li,Folawiyo Campbell-Ajala,Daniel Toyama,Robert Berry,Divya Tyamagundlu,Timothy Lillicrap,Oriana Riva
2024-10-23
Abstract:Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device's system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at <a class="link-external link-http" href="http://github.com/google-research/android_world" rel="external noopener nofollow">this http URL</a>.
Artificial Intelligence,Machine Learning