ScreenAgent: A Vision Language Model-driven Computer Control Agent

Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, Qi Wang

2024-02-09

Abstract:Existing Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled directly by a trained LLM agent. Powered by the computer, we can hopefully build a more generalized agent to assist humans in various daily digital works. In this paper, we construct an environment for a Vision Language Model (VLM) agent to interact with a real computer screen. Within this environment, the agent can observe screenshots and manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard actions. We also design an automated control pipeline that includes planning, acting, and reflecting phases, guiding the agent to continuously interact with the environment and complete multi-step tasks. Additionally, we construct the ScreenAgent Dataset, which collects screenshots and action sequences when completing a variety of daily computer tasks. Finally, we trained a model, ScreenAgent, which achieved computer control capabilities comparable to GPT-4V and demonstrated more precise UI positioning capabilities. Our attempts could inspire further research on building a generalist LLM agent. The code is available at \url{https://github.com/niuzaisheng/ScreenAgent}.

Artificial Intelligence,Human-Computer Interaction,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper aims to address the issue of how to enable large language models (LLMs) to directly control a computer screen to complete complex tasks. Specifically, the paper proposes a system called ScreenAgent, which interacts with a real computer screen through a visual language model (VLM). In this environment, the agent can observe screenshots and manipulate the graphical user interface (GUI) by outputting mouse and keyboard operations. Additionally, the paper designs an automated control process that includes three stages: planning, execution, and reflection, guiding the agent to continuously interact with the environment and complete multi-step tasks. To validate the effectiveness of this approach, the researchers constructed the ScreenAgent dataset and tested multiple models. The results show that ScreenAgent excels in precise UI localization capabilities, with performance comparable to GPT-4V. This work lays the foundation for further research on general LLM agents.