Abstract:Multi-modal AI systems will likely become a ubiquitous presence in our everyday lives. A promising approach to making these systems more interactive is to embody them as agents within physical and virtual environments. At present, systems leverage existing foundation models as the basic building blocks for the creation of embodied agents. Embedding agents within such environments facilitates the ability of models to process and interpret visual and contextual data, which is critical for the creation of more sophisticated and context-aware AI systems. For example, a system that can perceive user actions, human behavior, environmental objects, audio expressions, and the collective sentiment of a scene can be used to inform and direct agent responses within the given environment. To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can produce meaningful embodied actions. In particular, we explore systems that aim to improve agents based on next-embodied action prediction by incorporating external knowledge, multi-sensory inputs, and human feedback. We argue that by developing agentic AI systems in grounded environments, one can also mitigate the hallucinations of large foundation models and their tendency to generate environmentally incorrect outputs. The emerging field of Agent AI subsumes the broader embodied and agentic aspects of multimodal interactions. Beyond agents acting and interacting in the physical world, we envision a future where people can easily create any virtual reality or simulated scene and interact with agents embodied within the virtual environment.

Developing Embodied Multisensory Dialogue Agents

Beyond robotic speech: mutual benefits to cognitive psychology and artificial intelligence from the study of multimodal communication

Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

Agent AI: Surveying the Horizons of Multimodal Interaction

Introducing Brain-like Concepts to Embodied Hand-crafted Dialog Management System

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Language Acquisition is Embodied, Interactive, Emotive: a Research Proposal

“Artificial humans”: Psychology and neuroscience perspectives on embodiment and nonverbal communication

Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents

Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions

Mutual influence between language and perception in multi-agent communication games

Virtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Multimodal Grounding for Embodied AI via Augmented Reality Headsets for Natural Language Driven Task Planning

Crossmodal Language Grounding in an Embodied Neurocognitive Model

Embodied Artificial Intelligence through Distributed Adaptive Control: An Integrated Framework

Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community

Multimodal Human Machine Interactions in Virtual and Augmented Reality.

Seeing eye to eye: trustworthy embodiment for task-based conversational agents

Beyond Words: An Intelligent Human-Machine Dialogue System with Multimodal Generation and Emotional Comprehension

The promise and peril of interactive embodied agents for studying non-verbal communication: a machine learning perspective