Abstract:GUIs have long been central to human-computer interaction, providing an intuitive and visually-driven way to access and interact with digital systems. The advent of LLMs, particularly multimodal models, has ushered in a new era of GUI automation. They have demonstrated exceptional capabilities in natural language understanding, code generation, and visual processing. This has paved the way for a new generation of LLM-brained GUI agents capable of interpreting complex GUI elements and autonomously executing actions based on natural language instructions. These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversational commands. Their applications span across web navigation, mobile app interactions, and desktop automation, offering a transformative user experience that revolutionizes how individuals interact with software. This emerging field is rapidly advancing, with significant progress in both research and industry. To provide a structured understanding of this trend, this paper presents a comprehensive survey of LLM-brained GUI agents, exploring their historical evolution, core components, and advanced techniques. We address research questions such as existing GUI agent frameworks, the collection and utilization of data for training specialized GUI agents, the development of large action models tailored for GUI tasks, and the evaluation metrics and benchmarks necessary to assess their effectiveness. Additionally, we examine emerging applications powered by these agents. Through a detailed analysis, this survey identifies key research gaps and outlines a roadmap for future advancements in the field. By consolidating foundational knowledge and state-of-the-art developments, this work aims to guide both researchers and practitioners in overcoming challenges and unlocking the full potential of LLM-brained GUI agents.

Large Language User Interfaces: Voice Interactive User Interfaces powered by LLMs

Large Language Models Meet User Interfaces: The Case of Provisioning Feedback

Effort and Size Estimation in Software Projects with Large Language Model-based Intelligent Interfaces

User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants

Understanding User Experience in Large Language Model Interactions

Low-code LLM: Graphical User Interface over Large Language Models

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

Large Language Model-Brained GUI Agents: A Survey

The Importance of Understanding Language in Large Language Models

Making Large Language Models Interactive: A Pioneer Study on Supporting Complex Information-Seeking Tasks with Implicit Constraints

Broadening Access to Simulations for End-Users via Large Language Models: Challenges and Opportunities

Exploring the Potential of Large Language Models in Radiological Imaging Systems: Improving User Interface Design and Functional Capabilities

The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

Interpreting and learning voice commands with a Large Language Model for a robot system

Enabling Conversational Interaction with Mobile UI using Large Language Models

VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots

Persuasion Games using Large Language Models

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

Large Language Models Humanize Technology

Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles