Abstract:As deep learning technology research continues to progress, artificial intelligence technology is gradually empowering various fields. To achieve a more natural human-computer interaction experience, how to accurately recognize emotional state of speech interactions has become a new research hotspot. Sequence modeling methods based on deep learning techniques have promoted the development of emotion recognition, but the mainstream methods still suffer from insufficient multimodal information interaction, difficulty in learning emotion-related features, and low recognition accuracy. In this article, we propose a transformer-based deep-scale fusion network (TDFNet) for multimodal emotion recognition, solving the aforementioned problems. The multimodal embedding (ME) module in TDFNet uses pretrained models to alleviate the data scarcity problem by providing a priori knowledge of multimodal information to the model with the help of a large amount of unlabeled data. Furthermore, a mutual transformer (MT) module is introduced to learn multimodal emotional commonality and speaker-related emotional features to improve contextual emotional semantic understanding. In addition, we design a novel emotion feature learning method named the deep-scale transformer (DST), which further improves emotion recognition by aligning multimodal features and learning multiscale emotion features through GRUs with shared weights. To comparatively evaluate the performance of TDFNet, experiments are conducted with the IEMOCAP corpus under three reasonable data splitting strategies. The experimental results show that TDFNet achieves 82.08% WA and 82.57% UA in RA data splitting, which leads to 1.78% WA and 1.17% UA improvements over the previous state-of-the-art method, respectively. Benefiting from the attentively aligned mutual correlations and fine-grained emotion-related features, TDFNet successfully achieves significant improvements in multimodal emotion recognition.

Multi-Feature Fusion Based Deep Network for Image Semantic Recognition

A Multi-feature Fusion and SSAE-Based Deep Network for Image Semantic Recognition

Recognizing Image Semantic Information Through Multi-Feature Fusion and SSAE-Based Deep Network

Multi-modal Facial Expression Feature Based on Deep-Neural Networks

Emotion Recognition via Environmental Context and Human Body

Combining Multimodal Features Within A Fusion Network For Emotion Recognition In The Wild

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Using Scene and Semantic Features for Multi-modal Emotion Recognition

Image Emotion Recognition Based on Deep Neural Network

Affective Image Recognition with Multi-Attribute Knowledge in Deep Neural Networks

Deep Spatial-Temporal Feature Fusion for Facial Expression Recognition in Static Images

Multi-modal fusion network with complementarity and importance for emotion recognition

Learning Multi-level Representations for Image Emotion Recognition in the Deep Convolutional Network

Deep Fusion of Multi-Channel Neurophysiological Signal for Emotion Recognition and Monitoring

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

STM-Net Based Spatial-Temporal Multi-Modal Fusion Network for Emotion Recognition

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

Feature Discrimination and Diversification for Image Emotion Recognition

Two Feature Fusion Network Based on Efficient Deep Residual Shrinkage Spatial-Temporal Graph Convolution for Emotion Recognition from Gaits

EEG-based Emotion Recognition with Feature Fusion Networks.

TDFNet: Transformer-Based Deep-Scale Fusion Network for Multimodal Emotion Recognition