Abstract:Multimodal sentiment analysis holds significant importance within the realm of human-computer interaction. Due to the ease of collecting unlabeled online resources compared to the high costs associated with annotation, it becomes imperative for researchers to develop semi-supervised methods that leverage unlabeled data to enhance model performance. Existing semi-supervised approaches, particularly those applied to trivial image classification tasks, are not suitable for multimodal regression tasks due to their reliance on task-specific augmentation and thresholds designed for classification tasks. To address this limitation, we propose the Multimodal Consistency-based Teacher (MC-Teacher), which incorporates consistency-based pseudo-label technique into semi-supervised multimodal sentiment analysis. In our approach, we first propose synergistic consistency assumption which focus on the consistency among bimodal representation. Building upon this assumption, we develop a learnable filter network that autonomously learns how to identify misleading instances instead of threshold-based methods. This is achieved by leveraging both the implicit discriminant consistency on unlabeled instances and the explicit guidance on constructed training data with labeled instances. Additionally, we design the self-adaptive exponential moving average strategy to decouple the student and teacher networks, utilizing a heuristic momentum coefficient. Through both quantitative and qualitative experiments on two benchmark datasets, we demonstrate the outstanding performances of the proposed MC-Teacher approach. Furthermore, detailed analysis experiments and case studies are provided for each crucial component to intuitively elucidate the inner mechanism and further validate their effectiveness.

Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Semi-Supervised Multi-Modal Learning with Incomplete Modalities

Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach

Comprehensive Semi-Supervised Multi-Modal Learning.

Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis

A Fine-Grained Modal Label-Based Multi-Stage Network for Multimodal Sentiment Analysis.

Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning

Multi-modal Sentiment and Emotion Joint Analysis with a Deep Attentive Multi-task Learning Model

Dual-View Multimodal Interaction in Multimodal Sentiment Analysis

MixLab: An Informative Semi-supervised Method for Multi-label Classification

Kernel Semi-Supervised Graph Embedding Model for Multimodal and Mixmodal Data.

An Autoencoder-based Self-Supervised Learning for Multimodal Sentiment Analysis

Semi-supervised Sentiment Classification Based on Auxiliary Task Learning.

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

Learning Speaker-Independent Multimodal Representation for Sentiment Analysis

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Multi‐level Deep Correlative Networks for Multi‐modal Sentiment Analysis

Multimodal Consistency-based Teacher for Semi-supervised Multimodal Sentiment Analysis

Multi-Modal Multi-Label Semantic Indexing Of Images Based On Hybrid Ensemble Learning