Abstract:In this paper, we extensively present our solutions for the MuSe-Stress sub-challenge and the MuSe-Physio sub-challenge of Multimodal Sentiment Challenge (MuSe) 2021. The goal of MuSe-Stress sub-challenge is to predict the level of emotional arousal and valence in a time-continuous manner from audio-visual recordings and the goal of MuSe-Physio sub-challenge is to predict the level of psycho-physiological arousal from a) human annotations fused with b) galvanic skin response (also known as Electrodermal Activity (EDA)) signals from the stressed people. The Ulm-TSST dataset which is a novel subset of the audio-visual textual Ulm-Trier Social Stress dataset that features German speakers in a Trier Social Stress Test (TSST) induced stress situation is used in both sub-challenges. For the MuSe-Stress sub-challenge, we highlight our solutions in three aspects: 1) the audio-visual features and the bio-signal features are used for emotional state recognition. 2) the Long Short-Term Memory (LSTM) with the self-attention mechanism is utilized to capture complex temporal dependencies within the feature sequences. 3) the late fusion strategy is adopted to further boost the model's recognition performance by exploiting complementary information scattered across multimodal sequences. Our proposed model achieves CCC of 0.6159 and 0.4609 for valence and arousal respectively on the test set, which both rank in the top 3. For the MuSe-Physio sub-challenge, we first extract the audio-visual features and the bio-signal features from multiple modalities. Then, the LSTM module with the self-attention mechanism, and the Gated Convolutional Neural Networks (GCNN) as well as the LSTM network are utilized for modeling the complex temporal dependencies in the sequence. Finally, the late fusion strategy is used. Our proposed method also achieves CCC of 0.5412 on the test set, which ranks in the top 3.

Social Perception Prediction for MuSe 2024: Joint Learning of Multiple Perceptions

Going Beyond Multi-Task Dense Prediction with Synergy Embedding Models

The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition

Multimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model

Exclusive Modeling for MuSe-Personalisation Challenge

Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism

Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis

Exploiting Diverse Feature for Multimodal Sentiment Analysis

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

Multimodal Sentiment Analysis Based on Recurrent Neural Network and Multimodal Attention.

Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation

Temporal-aware Multimodal Feature Fusion for Sentiment Analysis

MuSe 2020 -- The First International Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop

Task-Oriented Mulsemedia Communication using Unified Perceiver and Conformal Prediction in 6G Metaverse Systems

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning

MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition

On the Consensus of Synchronous Temporal and Spatial Views: A Novel Multimodal Deep Learning Method for Social Video Prediction

Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning.