Abstract:GUIs have long been central to human-computer interaction, providing an intuitive and visually-driven way to access and interact with digital systems. The advent of LLMs, particularly multimodal models, has ushered in a new era of GUI automation. They have demonstrated exceptional capabilities in natural language understanding, code generation, and visual processing. This has paved the way for a new generation of LLM-brained GUI agents capable of interpreting complex GUI elements and autonomously executing actions based on natural language instructions. These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversational commands. Their applications span across web navigation, mobile app interactions, and desktop automation, offering a transformative user experience that revolutionizes how individuals interact with software. This emerging field is rapidly advancing, with significant progress in both research and industry. To provide a structured understanding of this trend, this paper presents a comprehensive survey of LLM-brained GUI agents, exploring their historical evolution, core components, and advanced techniques. We address research questions such as existing GUI agent frameworks, the collection and utilization of data for training specialized GUI agents, the development of large action models tailored for GUI tasks, and the evaluation metrics and benchmarks necessary to assess their effectiveness. Additionally, we examine emerging applications powered by these agents. Through a detailed analysis, this survey identifies key research gaps and outlines a roadmap for future advancements in the field. By consolidating foundational knowledge and state-of-the-art developments, this work aims to guide both researchers and practitioners in overcoming challenges and unlocking the full potential of LLM-brained GUI agents.

Construction of Semantic Model for GUI of Mobile Applications Using Deep Learning.

Close the Gap Between Deep Learning and Mobile Intelligence by Incorporating Training in the Loop

Deep Learning on Mobile and Embedded Devices: State-of-the-art, Challenges, and Future Directions

A Grey-Box Approach For Automated Gui-Model Generation Of Mobile Applications

Beyond Manual Modeling: Automating GUI Model Generation Using Design Documents

UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface

UI semantic component group detection: Grouping UI elements with similar semantics in mobile graphical user interface

Appaction: Automatic GUI Interaction for Mobile Apps Via Holistic Widget Perception

Production of Mobile English Language Teaching Application Based on Text Interface Using Deep Learning

Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcement Learning

A learning framework for semantic reach-to-grasp tasks integrating machine learning and optimization.

A reinforcement learning-based approach to testing GUI of moblie applications

Towards Better Semantic Understanding of Mobile Interfaces

Automated Reporting of GUI Design Violations for Mobile Apps

Practical Non-Intrusive GUI Exploration Testing with Visual-based Robotic Arms

Large Language Model-Brained GUI Agents: A Survey

Practical GUI Testing of Android Applications Via Model Abstraction and Refinement

Understanding Mobile GUI: from Pixel-Words to Screen-Sentences

Deep Learning Towards Mobile Applications

Humanoid: A Deep Learning-Based Approach to Automated Black-box Android App Testing.

MobileFlow: A Multimodal LLM For Mobile GUI Agent