Abstract:Currently, research on emotion recognition has shown that multi-modal data fusion has advantages in improving the accuracy and robustness of human emotion recognition, outperforming single-modal methods. Despite the promising results of existing methods, significant challenges remain in effectively fusing data from multiple modalities to achieve superior performance. Firstly, existing works tend to focus on generating a joint representation by fusing multi-modal data, with fewer methods considering the specific characteristics of each modality. Secondly, most methods fail to fully capture the intricate correlations among multiple modalities, often resorting to simplistic combinations of latent features. To address these challenges, we propose a novel fusion network for multi-modal emotion recognition. This network enhances the efficacy of multi-modal fusion while preserving the distinct characteristics of each modality. Specifically, a dual-stream multi-scale feature encoding (MFE) is designed to extract emotional information from both electroencephalogram (EEG) and peripheral physiological signals (PPS) temporal slices. Subsequently, a cross-modal global–local feature fusion module (CGFFM) is proposed to integrate global and local information from multi-modal data and then assign different importance to each modality, which makes the fusion data tend to the more important modalities. Meanwhile, the transformer module is employed to further learn the modality-specific information. Moreover, we introduce the adaptive collaboration block (ACB), which optimally leverages both modality-specific and cross-modality relations for enhanced integration and feature representation. Following extensive experiments on the DEAP and DREAMER multimodal datasets, our model achieves state-of-the-art performance.

DGSNet: Dual Graph Structure Network for Emotion Recognition in Multimodal Conversations

MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations

MMGCN: Multimodal Fusion Via Deep Graph Convolution Network for Emotion Recognition in Conversation

MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation

GraphMFT: A Graph Network based Multimodal Fusion Technique for Emotion Recognition in Conversation

GA2MIF: Graph and Attention Based Two-Stage Multi-Source Information Fusion for Conversational Emotion Detection

Bi-stream graph learning based multimodal fusion for emotion recognition in conversation

Dynamic Graph Neural Ordinary Differential Equation Network for Multi-modal Emotion Recognition in Conversation

A Novel and Powerful Dual-Stream Multi-Level Graph Convolution Network for Emotion Recognition

Dense Graph Convolutional with Joint Cross-Attention Network for Multimodal Emotion Recognition

DGFN Multimodal Emotion Analysis Model Based on Dynamic Graph Fusion Network

Fusion with Hierarchical Graphs for Multimodal Emotion Recognition

Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognition

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition

A Dual Attention-based Modality-Collaborative Fusion Network for Emotion Recognition

MLGAT: multi-layer graph attention networks for multimodal emotion recognition in conversations

Conversational emotion recognition studies based on graph convolutional neural networks and a dependent syntactic analysis