Abstract:Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities. However, the breadth of progress in multimodal research has made it difficult to identify the common themes and open questions in the field. By synthesizing a broad range of application domains and theoretical frameworks from both historical and recent perspectives, this paper is designed to provide an overview of the computational and theoretical foundations of multimodal machine learning. We start by defining three key principles of modality heterogeneity , connections , and interactions that have driven subsequent innovations, and propose a taxonomy of six core technical challenges: representation , alignment , reasoning , generation , transference , and quantification covering historical and recent trends. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches. We end by motivating several open problems for future research as identified by our taxonomy.

Multimodal Grounding for Language Processing

Grounded and Well-rounded: A Methodological Approach to the Study of Cross-modal and Cross-lingual Grounding

Presentation and Analysis of a Multimodal Dataset for Grounded Language Learning

Visual Grounding Strategies for Text-Only Natural Language Processing

Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents

GroundingGPT:Language Enhanced Multi-modal Grounding Model

Grounding Multimodal Large Language Models in Actions

Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Units

A Multimodal Connectionist Architecture for Unsupervised Grounding of Spatial Language

Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Visual Grounding With Joint Multimodal Representation and Interaction

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Multimodal Large Language Models: A Survey

What is Multimodality?

Multimodal Machine Learning: A Survey and Taxonomy

Towards Harnessing Large Language Models for Comprehension of Conversational Grounding

Experience Grounds Language

Crossmodal Language Grounding in an Embodied Neurocognitive Model

Grounding Language Models to Images for Multimodal Inputs and Outputs

A multimodal parallel architecture: A cognitive framework for multimodal interactions