Abstract:We present AutoGLM, a new series in the ChatGLM family, designed to serve as foundation agents for autonomous control of digital devices through Graphical User Interfaces (GUIs). While foundation models excel at acquiring human knowledge, they often struggle with decision-making in dynamic real-world environments, limiting their progress toward artificial general intelligence. This limitation underscores the importance of developing foundation agents capable of learning through autonomous environmental interactions by reinforcing existing models. Focusing on Web Browser and Phone as representative GUI scenarios, we have developed AutoGLM as a practical foundation agent system for real-world GUI interactions. Our approach integrates a comprehensive suite of techniques and infrastructures to create deployable agent systems suitable for user delivery. Through this development, we have derived two key insights: First, the design of an appropriate "intermediate interface" for GUI control is crucial, enabling the separation of planning and grounding behaviors, which require distinct optimization for flexibility and accuracy respectively. Second, we have developed a novel progressive training framework that enables self-evolving online curriculum reinforcement learning for AutoGLM. Our evaluations demonstrate AutoGLM's effectiveness across multiple domains. For web browsing, AutoGLM achieves a 55.2% success rate on VAB-WebArena-Lite (improving to 59.1% with a second attempt) and 96.2% on OpenTable evaluation tasks. In Android device control, AutoGLM attains a 36.2% success rate on AndroidLab (VAB-Mobile) and 89.7% on common tasks in popular Chinese APPs.

IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agents

GriddlyJS: A Web IDE for Reinforcement Learning

GridToPix: Training Embodied Agents with Minimal Supervision

AI Safety Gridworlds

Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks

NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds

GLIB: Efficient Exploration for Relational Model-Based Reinforcement Learning via Goal-Literal Babbling

GridMM: Grid Memory Map for Vision-and-Language Navigation

XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX

Situated Dialogue Learning through Procedural Environment Generation

Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments

Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents

Grid-wise control for multi-agent reinforcement learning in video game ai

Interactive Grounded Language Acquisition and Generalization in a 2D World

GenRL: Multimodal-foundation world models for generalization in embodied agents

AutoGLM: Autonomous Foundation Agents for GUIs

ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting

UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI

V-IRL: Grounding Virtual Intelligence in Real Life

Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots