Abstract:One of the major obstacles that has to be faced when applying automatic emotion recognition to realistic human-machine interaction systems is the scarcity of labeled data for training a robust model. Motivated by this concern, this paper seeks to utmost exploit unlabeled data that are pervasively available in the real-world and easy to be collected, by means of novel semi-supervised learning (SSL) approaches. Conventional SSL methods such as self-training, suffer from their inherent drawback of error accumulation, i.e., the samples that are misclassified by the system are continuously employed to train the model in the following learning iterations. To address this major issue, we first propose an enhanced learning strategy, by which we re-evaluate the previously automatically labeled samples in each learning iteration, in order to update the training set by correcting the mislabeled samples. We further exploit multiple modalities and models in the SSL system, by using collaborative SSL, where all modalities and models are considered simultaneously; samples are selected by means of minimizing the joint entropy. This strategy is supposed to not only improve the performance of the model for data annotation and consequently enhance the trustability of the automatically labeled data, but also to elevate the diversity of selected data. To evaluate the effectiveness of the proposed approaches, we performed extensive experiments on the remote collaborative and affective database, which includes multimodal recordings of spontaneous affective interactions of dyads. The empirical results show that the proposed approaches significantly outperform recently well-established SSL methods.

Semi-Supervised Multimodal Emotion Recognition with Class-Balanced Pseudo-labeling.

Leveraging Contrastive Learning and Self-Training for Multimodal Emotion Recognition with Limited Labeled Samples

Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning.

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning

Mining High-quality Samples from Raw Data and Majority Voting Method for Multimodal Emotion Recognition

Improving Multimodal Emotion Recognition by Leveraging Acoustic Adaptation and Visual Alignment

Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout

Leveraging Label Information for Multimodal Emotion Recognition

Multimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model

MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition

Audio-Guided Fusion Techniques for Multimodal Emotion Analysis

Speech Emotion Recognition by Combining a Unified First-Order Attention Network with Data Balance

Label Distribution Adaptation for Multimodal Emotion Recognition with Multi-label Learning

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

Leveraging Unlabeled Data for Emotion Recognition With Enhanced Collaborative Semi-Supervised Learning.

Multimodal Emotion Recognition in Noisy Environment Based on Progressive Label Revision

First-order Multi-label Learning with Cross-modal Interactions for Multimodal Emotion Recognition

Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition