Abstract:Currently, research on emotion recognition has shown that multi-modal data fusion has advantages in improving the accuracy and robustness of human emotion recognition, outperforming single-modal methods. Despite the promising results of existing methods, significant challenges remain in effectively fusing data from multiple modalities to achieve superior performance. Firstly, existing works tend to focus on generating a joint representation by fusing multi-modal data, with fewer methods considering the specific characteristics of each modality. Secondly, most methods fail to fully capture the intricate correlations among multiple modalities, often resorting to simplistic combinations of latent features. To address these challenges, we propose a novel fusion network for multi-modal emotion recognition. This network enhances the efficacy of multi-modal fusion while preserving the distinct characteristics of each modality. Specifically, a dual-stream multi-scale feature encoding (MFE) is designed to extract emotional information from both electroencephalogram (EEG) and peripheral physiological signals (PPS) temporal slices. Subsequently, a cross-modal global–local feature fusion module (CGFFM) is proposed to integrate global and local information from multi-modal data and then assign different importance to each modality, which makes the fusion data tend to the more important modalities. Meanwhile, the transformer module is employed to further learn the modality-specific information. Moreover, we introduce the adaptive collaboration block (ACB), which optimally leverages both modality-specific and cross-modality relations for enhanced integration and feature representation. Following extensive experiments on the DEAP and DREAMER multimodal datasets, our model achieves state-of-the-art performance.

A Two-Stage Multi-Modal Multi-Label Emotion Recognition Decision System Based on GCN

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

A Two-Stage Multimodal Emotion Recognition Model Based on Graph Contrastive Learning

A Novel and Powerful Dual-Stream Multi-Level Graph Convolution Network for Emotion Recognition

Emotion Recognition Model Based on Multimodal Decision Fusion

Multi-head attention fusion networks for multi-modal speech emotion recognition

GA2MIF: Graph and Attention Based Two-Stage Multi-Source Information Fusion for Conversational Emotion Detection

Multi-Modality Emotion Recognition Model with GAT-Based Multi-Head Inter-Modality Attention

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

A multi-stage dynamical fusion network for multimodal emotion recognition

Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing

Semi-supervised Bayesian Deep Multi-modal Emotion Recognition.

Correlation-Driven Multi-Modality Graph Decomposition for Cross-Subject Emotion Recognition

Multi-modal fusion network with complementarity and importance for emotion recognition

MLGAT: multi-layer graph attention networks for multimodal emotion recognition in conversations

Multi-modal Emotion Recognition Based on Speech and Image.

Multi-modal Emotion Recognition Based on Deep Learning in Speech, Video and Text

MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Affect-GCN: a multimodal graph convolutional network for multi-emotion with intensity recognition and sentiment analysis in dialogues