Abstract:Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots. While there has been an explosion of interest in multimodal learning, these methods are focused on a small set of modalities primarily in language, vision, and audio. In order to accelerate generalization towards diverse and understudied modalities, this paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. Since adding new models for every new modality becomes prohibitively expensive, a critical technical challenge is heterogeneity quantification: how can we measure which modalities encode similar information and interactions in order to permit parameter sharing with previous modalities? This paper proposes two new information theoretic metrics for heterogeneity quantification: (1) modality heterogeneity studies how similar 2 modalities {X1,X2} are by measuring how much information can be transferred from X1 to X2, while (2) interaction heterogeneity studies how similarly pairs of modalities {X1,X2}, {X3,X4} interact by measuring how much information can be transferred from fusing {X1,X2} to {X3,X4}. We show the importance of these 2 proposed metrics as a way to automatically prioritize the fusion of modalities that contain unique information or interactions. The result is a single model, HighMMT, that scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas. Not only does HighMMT outperform prior methods on the tradeoff between performance and efficiency, it also demonstrates a crucial scaling behavior: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning.

xTML: A Unified Heterogeneous Transfer Metric Learning Framework for Multimedia Applications [Application Notes]

Online latent semantic hashing for cross-media retrieval.

Discrete Cross-Modal Hashing for Efficient Multimedia Retrieval

Multi-view Heterogeneous Fusion and Embedding for Categorical Attributes on Mixed Data.

Online Heterogeneous Transfer Metric Learning.

General Heterogeneous Transfer Distance Metric Learning Via Knowledge Fragments Transfer.

Heterogeneous Multitask Metric Learning Across Multiple Domains

On Combining Side Information and Unlabeled Data for Heterogeneous Multi-Task Metric Learning.

Heterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval.

Distance Metric Learning from Uncertain Side Information with Application to Automated Photo Tagging

Deep Heterogeneous Multi-Task Metric Learning for Visual Recognition and Retrieval

Toward Effortless TV-to-Online (T2O) Experience: A Novel Metric Learning Approach.

Toward Intelligent Product Retrieval for TV-to-Online (T2O) Application: A Transfer Metric Learning Approach

A unified multimodal classification framework based on deep metric learning

Transfer Metric Learning: Algorithms, Applications and Outlooks.

Transferring Knowledge Fragments for Learning Distance Metric from A Heterogeneous Domain

Decomposition-based transfer distance metric learning for image classification.

Exploiting High-Order Information in Heterogeneous Multi-Task Feature Learning.

Distance Metric Facilitated Transportation Between Heterogeneous Domains.

Multi-component transfer metric learning for handling unrelated source domain samples

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning