Abstract:Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities. The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities. However, existing work learns joint representations by requiring all modalities as input and as a result, the learned representations may be sensitive to noisy or missing modalities at test time. With the recent success of sequence to sequence (Seq2Seq) models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time. In this paper, we propose a method to learn robust joint representations by translating between modalities. Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input. We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities. Once our translation model is trained with paired multimodal data, we only need data from the source modality at test time for final sentiment prediction. This ensures that our model remains robust from perturbations or missing information in the other modalities. We train our model with a coupled translation-prediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI, ICT-MMMO, and YouTube. Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to missing or perturbed modalities.

Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Using Bilingual Knowledge and Ensemble Techniques for Unsupervised Chinese Sentiment Analysis.

Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities

Learning Bilingual Embedding Model for Cross-Language Sentiment Classification

Learning Sentiment Memories for Sentiment Modification Without Parallel Data.

Scalable Sentiment for Sequence-to-sequence Chatbot Response with Performance Analysis

Towards Fine-grained Text Sentiment Transfer

Specificity-Driven Cascading Approach for Unsupervised Sentiment Modification

Unsupervised Self-Training for Sentiment Analysis of Code-Switched Data

Bilingual co-training for sentiment classification of chinese product reviews

A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer

A Multi-task Learning Framework for Evaluating Machine Translation of Emotion-loaded User-generated Content

Joint Bilingual Sentiment Classification with Unlabeled Parallel Corpora.

Investigation of Sentiment Controllable Chatbot

Cross-Domain Sentiment Encoding through Stochastic Word Embedding

Unpaired Multimodal Neural Machine Translation via Reinforcement Learning

Reinforcement learning in sentiment analysis: a review and future directions

Deep Learning Paradigm with Transformed Monolingual Word Embeddings for Multilingual Sentiment Analysis

Sentiment Analysis through LLM Negotiations

Transfer Learning for Sentiment Classification Using Bidirectional Encoder Representations from Transformers (BERT) Model