Abstract:As AI continues to advance, there is a growing demand for systems that go beyond language-based assistance and move toward intelligent agents capable of performing real-world actions. This evolution requires the transition from traditional Large Language Models (LLMs), which excel at generating textual responses, to Large Action Models (LAMs), designed for action generation and execution within dynamic environments. Enabled by agent systems, LAMs hold the potential to transform AI from passive language understanding to active task completion, marking a significant milestone in the progression toward artificial general intelligence. In this paper, we present a comprehensive framework for developing LAMs, offering a systematic approach to their creation, from inception to deployment. We begin with an overview of LAMs, highlighting their unique characteristics and delineating their differences from LLMs. Using a Windows OS-based agent as a case study, we provide a detailed, step-by-step guide on the key stages of LAM development, including data collection, model training, environment integration, grounding, and evaluation. This generalizable workflow can serve as a blueprint for creating functional LAMs in various application domains. We conclude by identifying the current limitations of LAMs and discussing directions for future research and industrial deployment, emphasizing the challenges and opportunities that lie ahead in realizing the full potential of LAMs in real-world applications. The code for the data collection process utilized in this paper is publicly available at: <a class="link-external link-https" href="https://github.com/microsoft/UFO/tree/main/dataflow" rel="external noopener nofollow">this https URL</a>, and comprehensive documentation can be found at <a class="link-external link-https" href="https://microsoft.github.io/UFO/dataflow/overview/" rel="external noopener nofollow">this https URL</a>.

Large Language Model Based Multi-Agent System Augmented Complex Event Processing Pipeline for Internet of Multimedia Things

Large Language Model based Multi-Agents: A Survey of Progress and Challenges

OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer

Self-Adaptive Large Language Model (LLM)-Based Multiagent Systems

Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

LLM Harmony: Multi-Agent Communication for Problem Solving

MindAgent: Emergent Gaming Interaction

Synergistic Simulations: Multi-Agent Problem Solving with Large Language Models

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

Large Language Model-Driven Cross-Domain Orchestration Using Multi-Agent Workflow

Building Cooperative Embodied Agents Modularly with Large Language Models

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

Exploring Autonomous Agents through the Lens of Large Language Models: A Review

Large Multimodal Agents: A Survey

LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead

Navigating Complexity: Orchestrated Problem Solving with Multi-Agent LLMs

MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents

Large Action Models: From Inception to Implementation

Large Language Model Evaluation Via Multi AI Agents: Preliminary results

LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulation

Chain of Agents: Large Language Models Collaborating on Long-Context Tasks