Abstract:In the field of Multimodal Sentiment Analysis (MSA), the prevailing methods are devoted to developing intricate network architectures to capture the intra- and inter-modal dynamics, which necessitates numerous parameters and poses more difficulties in terms of interpretability in multimodal modeling. Besides, the heterogeneous nature of multiple modalities (text, audio, and vision) introduces significant modality gaps, thereby making multimodal representation learning an ongoing challenge. To address the aforementioned issues, by considering the learning process of modalities as multiple subtasks, we propose a novel approach named Multi-Task Momentum Distillation (MTMD) which succeeds in reducing the gap among different modalities. Specifically, according to the abundance of semantic information, we treat the subtasks of textual and multimodal representations as the teacher networks while the subtasks of acoustic and visual representations as the student ones to present knowledge distillation, which transfers the sentiment-related knowledge guided by the regression and classification subtasks. Additionally, we adopt unimodal momentum models to explore modality-specific knowledge deeply and employ adaptive momentum fusion factors to learn a robust multimodal representation. Furthermore, we provide a theoretical perspective of mutual information maximization by interpreting MTMD as generating sentiment-related views in various ways. Extensive experiments illustrate the superiority of our approach compared with the state-of-the-art methods in MSA.

Multimodal sentiment analysis with two-phase multi-task learning

Multimodal Sentiment Analysis With Two-Phase Multi-Task Learning

A text guided multi-task learning network for multimodal sentiment analysis

M$^{3}$SA: Multimodal Sentiment Analysis Based on Multi-Scale Feature Extraction and Multi-Task Learning

Low-rank tensor fusion and self-supervised multi-task multimodal sentiment analysis

Text-oriented Modality Reinforcement Network for Multimodal Sentiment Analysis from Unaligned Multimodal Sequences

Multi-modal Sentiment and Emotion Joint Analysis with a Deep Attentive Multi-task Learning Model

Multi-Task Momentum Distillation for Multimodal Sentiment Analysis

Multi-layer cross-modality attention fusion network for multimodal sentiment analysis

Dynamic Weighted Multitask Learning and Contrastive Learning for Multimodal Sentiment Analysis

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

Towards Robust Multimodal Sentiment Analysis with Incomplete Data

VLP2MSA: Expanding Vision-Language Pre-Training to Multimodal Sentiment Analysis

Multimodal Sentiment Recognition With Multi-Task Learning

Sentiment-aware Multimodal Pre-Training for Multimodal Sentiment Analysis

Tri-CLT: Learning Tri-Modal Representations with Contrastive Learning and Transformer for Multimodal Sentiment Recognition

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis