Abstract:Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities. The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities. However, existing work learns joint representations by requiring all modalities as input and as a result, the learned representations may be sensitive to noisy or missing modalities at test time. With the recent success of sequence to sequence (Seq2Seq) models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time. In this paper, we propose a method to learn robust joint representations by translating between modalities. Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input. We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities. Once our translation model is trained with paired multimodal data, we only need data from the source modality at test time for final sentiment prediction. This ensures that our model remains robust from perturbations or missing information in the other modalities. We train our model with a coupled translation-prediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI, ICT-MMMO, and YouTube. Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to missing or perturbed modalities.

Variational Multimodal Machine Translation with Underlying Semantic Alignment

Multimodal Image-to-Image Translation via Mutual Information Estimation and Maximization

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Latent Variable Model for Multi-modal Translation

Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Enhancing Neural Machine Translation with Dual-Side Multimodal Awareness

Bilingual–Visual Consistency for Multimodal Neural Machine Translation

A Joint Learning Model with Variational Interaction for Multilingual Program Translation

A Visual Attention Grounding Neural Model for Multimodal Machine Translation

Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

Video Pivoting Unsupervised Multi-Modal Machine Translation

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models

Visual Agreement Regularized Training for Multi-Modal Machine Translation

Multimodal Transformer For Multimodal Machine Translation

Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets

Visual Topic Semantic Enhanced Machine Translation for Multi-Modal Data Efficiency

Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities

Unified Generative and Discriminative Training for Multi-modal Large Language Models

EMMeTT: Efficient Multimodal Machine Translation Training