Abstract:As deep learning technology research continues to progress, artificial intelligence technology is gradually empowering various fields. To achieve a more natural human-computer interaction experience, how to accurately recognize emotional state of speech interactions has become a new research hotspot. Sequence modeling methods based on deep learning techniques have promoted the development of emotion recognition, but the mainstream methods still suffer from insufficient multimodal information interaction, difficulty in learning emotion-related features, and low recognition accuracy. In this article, we propose a transformer-based deep-scale fusion network (TDFNet) for multimodal emotion recognition, solving the aforementioned problems. The multimodal embedding (ME) module in TDFNet uses pretrained models to alleviate the data scarcity problem by providing a priori knowledge of multimodal information to the model with the help of a large amount of unlabeled data. Furthermore, a mutual transformer (MT) module is introduced to learn multimodal emotional commonality and speaker-related emotional features to improve contextual emotional semantic understanding. In addition, we design a novel emotion feature learning method named the deep-scale transformer (DST), which further improves emotion recognition by aligning multimodal features and learning multiscale emotion features through GRUs with shared weights. To comparatively evaluate the performance of TDFNet, experiments are conducted with the IEMOCAP corpus under three reasonable data splitting strategies. The experimental results show that TDFNet achieves 82.08% WA and 82.57% UA in RA data splitting, which leads to 1.78% WA and 1.17% UA improvements over the previous state-of-the-art method, respectively. Benefiting from the attentively aligned mutual correlations and fine-grained emotion-related features, TDFNet successfully achieves significant improvements in multimodal emotion recognition.

Transformer-Based Domain Adaptation for Multi-Modal Emotion Recognition in Response to Game Animation Videos.

Multi-view Domain-Adaptive Representation Learning for EEG-based Emotion Recognition

Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Multi-Source Domain Adaptation with Transformer-based Feature Generation for Subject-Independent EEG-based Emotion Recognition

Emotion Recognition Using Transformers with Masked Learning

Multi-Source EEG Emotion Recognition via Dynamic Contrastive Domain Adaptation

EmT: A Novel Transformer for Generalized Cross-subject EEG Emotion Recognition

Knowledge distillation based lightweight domain adversarial neural network for electroencephalogram-based emotion recognition

Adversarial Domain Generalized Transformer for Cross-Corpus Speech Emotion Recognition

Hybrid transfer learning strategy for cross-subject EEG emotion recognition

EEG-based Emotion Recognition Via Transformer Neural Architecture Search

Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

Non-Contact Emotion Recognition Combining Heart Rate and Facial Expression for Interactive Gaming Environments

Domain adversarial learning for emotion recognition

TDFNet: Transformer-Based Deep-Scale Fusion Network for Multimodal Emotion Recognition

A Lightweight Domain Adversarial Neural Network Based on Knowledge Distillation for EEG-based Cross-subject Emotion Recognition

Domain-adaptive emotion recognition based on horizontal vertical flow representation of EEG signals

Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition

Multi-source Selective Graph Domain Adaptation Network for cross-subject EEG emotion recognition