Abstract:As deep learning technology research continues to progress, artificial intelligence technology is gradually empowering various fields. To achieve a more natural human-computer interaction experience, how to accurately recognize emotional state of speech interactions has become a new research hotspot. Sequence modeling methods based on deep learning techniques have promoted the development of emotion recognition, but the mainstream methods still suffer from insufficient multimodal information interaction, difficulty in learning emotion-related features, and low recognition accuracy. In this article, we propose a transformer-based deep-scale fusion network (TDFNet) for multimodal emotion recognition, solving the aforementioned problems. The multimodal embedding (ME) module in TDFNet uses pretrained models to alleviate the data scarcity problem by providing a priori knowledge of multimodal information to the model with the help of a large amount of unlabeled data. Furthermore, a mutual transformer (MT) module is introduced to learn multimodal emotional commonality and speaker-related emotional features to improve contextual emotional semantic understanding. In addition, we design a novel emotion feature learning method named the deep-scale transformer (DST), which further improves emotion recognition by aligning multimodal features and learning multiscale emotion features through GRUs with shared weights. To comparatively evaluate the performance of TDFNet, experiments are conducted with the IEMOCAP corpus under three reasonable data splitting strategies. The experimental results show that TDFNet achieves 82.08% WA and 82.57% UA in RA data splitting, which leads to 1.78% WA and 1.17% UA improvements over the previous state-of-the-art method, respectively. Benefiting from the attentively aligned mutual correlations and fine-grained emotion-related features, TDFNet successfully achieves significant improvements in multimodal emotion recognition.

Frame-Transformer Emotion Classification Network.

A Multi-task Neural Approach for Emotion Attribution, Classification and Summarization

TDFNet: Transformer-Based Deep-Scale Fusion Network for Multimodal Emotion Recognition

EmT: A Novel Transformer for Generalized Cross-subject EEG Emotion Recognition

Modelling Temporal Information Using Discrete Fourier Transform for Recognizing Emotions in User-generated Videos

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

TC-Net: A Transformer Capsule Network for EEG-based emotion recognition

Emotional Video Captioning With Vision-Based Emotion Interpretation Network

FLDNet: Frame-Level Distilling Neural Network for EEG Emotion Recognition

CTNet: Conversational Transformer Network for Emotion Recognition

Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

A transformer-based approach to video frame-level prediction in Affective Behaviour Analysis In-the-wild

A Dual-Branch Dynamic Graph Convolution Based Adaptive TransFormer Feature Fusion Network for EEG Emotion Recognition

End-to-End Continuous Emotion Recognition from Video Using 3D Convlstm Networks

Audio-Visual Fusion Network Based on Conformer for Multimodal Emotion Recognition

MTLFuseNet: A novel emotion recognition model based on deep latent feature fusion of EEG signals and multi-task learning

Clip-aware expressive feature learning for video-based facial expression recognition

CSAT-FTCN: A Fuzzy-Oriented Model with Contextual Self-attention Network for Multimodal Emotion Recognition

Computational Integrative Physiology: At the Convergence of the Life and Computational Sciences

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Long Short Term Memory Recurrent Neural Network Based Encoding Method for Emotion Recognition in Video.