Abstract:Currently, research on emotion recognition has shown that multi-modal data fusion has advantages in improving the accuracy and robustness of human emotion recognition, outperforming single-modal methods. Despite the promising results of existing methods, significant challenges remain in effectively fusing data from multiple modalities to achieve superior performance. Firstly, existing works tend to focus on generating a joint representation by fusing multi-modal data, with fewer methods considering the specific characteristics of each modality. Secondly, most methods fail to fully capture the intricate correlations among multiple modalities, often resorting to simplistic combinations of latent features. To address these challenges, we propose a novel fusion network for multi-modal emotion recognition. This network enhances the efficacy of multi-modal fusion while preserving the distinct characteristics of each modality. Specifically, a dual-stream multi-scale feature encoding (MFE) is designed to extract emotional information from both electroencephalogram (EEG) and peripheral physiological signals (PPS) temporal slices. Subsequently, a cross-modal global–local feature fusion module (CGFFM) is proposed to integrate global and local information from multi-modal data and then assign different importance to each modality, which makes the fusion data tend to the more important modalities. Meanwhile, the transformer module is employed to further learn the modality-specific information. Moreover, we introduce the adaptive collaboration block (ACB), which optimally leverages both modality-specific and cross-modality relations for enhanced integration and feature representation. Following extensive experiments on the DEAP and DREAMER multimodal datasets, our model achieves state-of-the-art performance.

Branch-Fusion-Net for Multi-Modal Continuous Dimensional Emotion Recognition

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

A multimodal fusion-based deep learning framework combined with local-global contextual TCNs for continuous emotion recognition from videos

Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

An Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition

MRFER: Multi-Channel Robust Feature Enhanced Fusion for Multi-Modal Emotion Recognition

Multi-head attention fusion networks for multi-modal speech emotion recognition

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Multimodal Transformer Fusion for Continuous Emotion Recognition

Deep Emotional Arousal Network for Multimodal Sentiment Analysis and Emotion Recognition

Emotion recognition by fusing time synchronous and time asynchronous representations

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

Multi-modal fusion network with complementarity and importance for emotion recognition

A multi-stage dynamical fusion network for multimodal emotion recognition

MTLFuseNet: A novel emotion recognition model based on deep latent feature fusion of EEG signals and multi-task learning

GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognition

Residual multimodal Transformer for expression‐EEG fusion continuous emotion recognition