Abstract:GUIs have long been central to human-computer interaction, providing an intuitive and visually-driven way to access and interact with digital systems. The advent of LLMs, particularly multimodal models, has ushered in a new era of GUI automation. They have demonstrated exceptional capabilities in natural language understanding, code generation, and visual processing. This has paved the way for a new generation of LLM-brained GUI agents capable of interpreting complex GUI elements and autonomously executing actions based on natural language instructions. These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversational commands. Their applications span across web navigation, mobile app interactions, and desktop automation, offering a transformative user experience that revolutionizes how individuals interact with software. This emerging field is rapidly advancing, with significant progress in both research and industry. To provide a structured understanding of this trend, this paper presents a comprehensive survey of LLM-brained GUI agents, exploring their historical evolution, core components, and advanced techniques. We address research questions such as existing GUI agent frameworks, the collection and utilization of data for training specialized GUI agents, the development of large action models tailored for GUI tasks, and the evaluation metrics and benchmarks necessary to assess their effectiveness. Additionally, we examine emerging applications powered by these agents. Through a detailed analysis, this survey identifies key research gaps and outlines a roadmap for future advancements in the field. By consolidating foundational knowledge and state-of-the-art developments, this work aims to guide both researchers and practitioners in overcoming challenges and unlocking the full potential of LLM-brained GUI agents.

Poster: Enabling Agent-centric Interaction on Smartphones with LLM-based UI Reassembling

AppAgent: Multimodal Agents as Smartphone Users

Enabling Conversational Interaction with Mobile UI using Large Language Models

An Agent-Based Multimodal User Interfaces Middleware for Ubiquitous Computing Environments: Intelligent Exhibition Hall

Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

MobileAgent: enhancing mobile control via human-machine interaction and SOP integration

AppAgent v2: Advanced Agent for Flexible Mobile Interactions

Mobile assistant: enhancing desktop interaction using mobile phone

SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications

Comprehensive Cognitive LLM Agent for Smartphone GUI Automation

MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices

Mobile-Env: A Universal Platform for Training and Evaluation of Mobile Interaction

Large Language Model-Brained GUI Agents: A Survey

Human-Centered LLM-Agent User Interface: A Position Paper

ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents

CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

Interaction Proxy Manager: Semantic Model Generation and Run-time Support for Reconstructing Ubiquitous User Interfaces of Mobile Services.

Empowering LLM to use Smartphone for Intelligent Task Automation

Enabling On-Device LLMs Personalization with Smartphone Sensing

Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception