Abstract:Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots. While there has been an explosion of interest in multimodal learning, these methods are focused on a small set of modalities primarily in language, vision, and audio. In order to accelerate generalization towards diverse and understudied modalities, this paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. Since adding new models for every new modality becomes prohibitively expensive, a critical technical challenge is heterogeneity quantification: how can we measure which modalities encode similar information and interactions in order to permit parameter sharing with previous modalities? This paper proposes two new information theoretic metrics for heterogeneity quantification: (1) modality heterogeneity studies how similar 2 modalities {X1,X2} are by measuring how much information can be transferred from X1 to X2, while (2) interaction heterogeneity studies how similarly pairs of modalities {X1,X2}, {X3,X4} interact by measuring how much information can be transferred from fusing {X1,X2} to {X3,X4}. We show the importance of these 2 proposed metrics as a way to automatically prioritize the fusion of modalities that contain unique information or interactions. The result is a single model, HighMMT, that scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas. Not only does HighMMT outperform prior methods on the tradeoff between performance and efficiency, it also demonstrates a crucial scaling behavior: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning.

Quantification of Natural Multimodal Interaction Capacity

Modality Capacity and Appropriateness in Multimodal Display of Complex Non-Semantic Information Stream

Multimodal Fusion Interactions: A Study of Human and Automatic Quantification

Multimodal Systems: Taxonomy, Methods, and Challenges

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

Asymmetric Interaction Effects of Information with Varying Priorities in Visual Working Memory

Working Memory Capacity for Gesture-Command Associations in Gestural Interaction.

Review of Sensory Feedback Simulation Methods in Multi-modal Human-computer Interaction

Intelligence Methods of Multi-Modal Information Fusion in Human-Computer Interaction

Real-time multimodal interaction in virtual reality - a case study with a large virtual interface

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

A Review of Multimodal Interaction

Multimodal interaction: A review

The Moving-speed Capacity of Objects in Visual Working Memory

Intermodulation from Unisensory to Multisensory Perception: A Review

Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications

Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction

Multimodal integration-a statistical view

Distractive Effect of Multimodal Information in Multisensory Learning

Reducing the Cognitive Load of Playing a Digital Tabletop Game with a Multimodal Interface

Multimodal Interfaces for Effective Teleoperation