Abstract:Multimodal emotion analysis involves the integration of information from various modalities to better understand human emotions. In this paper, we propose the Cross-modal Emotion Recognition based on multi-layer semantic fusion (CM-MSF) model, which aims to leverage the complementarity of important information between modalities and extract advanced features in an adaptive manner. To achieve comprehensive and rich feature extraction from multimodal sources, considering different dimensions and depth levels, we design a parallel deep learning algorithm module that focuses on extracting features from individual modalities, ensuring cost-effective alignment of extracted features. Furthermore, a cascaded cross-modal encoder module based on Bidirectional Long Short-Term Memory (BILSTM) layer and Convolutional 1D (ConV1d) is introduced to facilitate inter-modal information complementation. This module enables the seamless integration of information across modalities, effectively addressing the challenges associated with signal heterogeneity. To facilitate flexible and adaptive information selection and delivery, we design the Mask-gated Fusion Networks (MGF-module), which combines masking technology with gating structures. This approach allows for precise control over the information flow of each modality through gating vectors, mitigating issues related to low recognition accuracy and emotional misjudgment caused by complex features and noisy redundant information. The CM-MSF model underwent evaluation using the widely recognized multimodal emotion recognition datasets CMU-MOSI and CMU-MOSEI. The experimental findings illustrate the exceptional performance of the model, with binary classification accuracies of 89.1% and 88.6%, as well as F1 scores of 87.9% and 88.1% on the CMU-MOSI and CMU-MOSEI datasets, respectively. These results unequivocally validate the effectiveness of our approach in accurately recognizing and classifying emotions.

Semantic Alignment Network for Multi-modal Emotion Recognition

Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

AMSA: Adaptive Multimodal Learning for Sentiment Analysis

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Research on cross-modal emotion recognition based on multi-layer semantic fusion

A Multi-Level Alignment and Cross-Modal Unified Semantic Graph Refinement Network for Conversational Emotion Recognition

Semantic Alignment for Multimodal Large Language Models

Multi-modal fusion network with complementarity and importance for emotion recognition

Multi-level Multi-task Representation Learning with Adaptive Fusion for Multimodal Sentiment Analysis

Multi-modal Siamese Network for Entity Alignment

MSAF: Multimodal Split Attention Fusion

AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations

Multi-head attention fusion networks for multi-modal speech emotion recognition

Semantic-specific multimodal relation learning for sentiment analysis

Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition

Deep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis.

Multi-layer cross-modality attention fusion network for multimodal sentiment analysis

A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition

Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignment

Multi-modal Attention for Speech Emotion Recognition