Abstract:Dynamic expression recognition in the wild is a challenging task due to various obstacles, including low light condition, non-positive face, and face occlusion. Purely vision-based approaches may not suffice to accurately capture the complexity of human emotions. To address this issue, we propose a Transformer-based Multimodal Emotional Perception (T-MEP) framework capable of effectively extracting multimodal information and achieving significant augmentation. Specifically, we design three transformer-based encoders to extract modality-specific features from audio, image, and text sequences, respectively. Each encoder is carefully designed to maximize its adaptation to the corresponding modality. In addition, we design a transformer-based multimodal information fusion module to model cross-modal representation among these modalities. The unique combination of self-attention and cross-attention in this module enhances the robustness of output-integrated features in encoding emotion. By mapping the information from audio and textual features to the latent space of visual features, this module aligns the semantics of the three modalities for cross-modal information augmentation. Finally, we evaluate our method on three popular datasets (MAFW, DFEW, and AFEW) through extensive experiments, which demonstrate its state-of-the-art performance. This research offers a promising direction for future studies to improve emotion recognition accuracy by exploiting the power of multimodal features.

DWFormer: Dynamic Window transFormer for Speech Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Hierarchical transformer speech depression detection model research based on Dynamic window and Attention merge

Attention-Enhanced Connectionist Temporal Classification for Discrete Speech Emotion Recognition

DST: Deformable Speech Transformer for Emotion Recognition

Multi-Scale Temporal Transformer For Speech Emotion Recognition

Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognition

SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speech

DropFormer: A Dynamic Noise-Dropping Transformer for Speech Emotion Recognition

Shift Window Based Framework for Emotional Change Detection of Speech

A Window Attention Based Transformer for Automatic Speech Recognition

MelTrans: Mel-Spectrogram Relationship-Learning for Speech Emotion Recognition via Transformers

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Temporal-spatial Representation Learning Transformer for EEG-based Emotion Recognition

EEMD-Based Speaker Automatic Emotional Recognition in Chinese Mandarin

Speech Emotion Recognition with Complementary Acoustic Representations.

Temporal-Frequency State Space Duality: An Efficient Paradigm for Speech Emotion Recognition

EigenEmo: Spectral Utterance Representation Using Dynamic Mode Decomposition for Speech Emotion Classification

HiTrans: A Transformer-Based Context- and Speaker-Sensitive Model for Emotion Detection in Conversations.

Automatic Emotion Variation Detection Using Multi-Scaled Sliding Window