Abstract:Multimodal emotion analysis involves the integration of information from various modalities to better understand human emotions. In this paper, we propose the Cross-modal Emotion Recognition based on multi-layer semantic fusion (CM-MSF) model, which aims to leverage the complementarity of important information between modalities and extract advanced features in an adaptive manner. To achieve comprehensive and rich feature extraction from multimodal sources, considering different dimensions and depth levels, we design a parallel deep learning algorithm module that focuses on extracting features from individual modalities, ensuring cost-effective alignment of extracted features. Furthermore, a cascaded cross-modal encoder module based on Bidirectional Long Short-Term Memory (BILSTM) layer and Convolutional 1D (ConV1d) is introduced to facilitate inter-modal information complementation. This module enables the seamless integration of information across modalities, effectively addressing the challenges associated with signal heterogeneity. To facilitate flexible and adaptive information selection and delivery, we design the Mask-gated Fusion Networks (MGF-module), which combines masking technology with gating structures. This approach allows for precise control over the information flow of each modality through gating vectors, mitigating issues related to low recognition accuracy and emotional misjudgment caused by complex features and noisy redundant information. The CM-MSF model underwent evaluation using the widely recognized multimodal emotion recognition datasets CMU-MOSI and CMU-MOSEI. The experimental findings illustrate the exceptional performance of the model, with binary classification accuracies of 89.1% and 88.6%, as well as F1 scores of 87.9% and 88.1% on the CMU-MOSI and CMU-MOSEI datasets, respectively. These results unequivocally validate the effectiveness of our approach in accurately recognizing and classifying emotions.

Adaptive Sensitive Information Recognition Based on Multimodal Information Inference in Social Networks

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

A multimodal sentiment recognition method based on attention mechanism

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

Social Media Public Opinion Detection Using Multimodal Natural Language Processing and Attention Mechanisms

Multi-Modal Fake News Detection on Social Media with Dual Attention Fusion Networks

Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion

Consumer Intention Recognition and Behavior Prediction of Social E-commerce Users Based on Multimodal Fusion

MIMF: Mutual Information-Driven Multimodal Fusion

Semantic‐enhanced multimodal fusion network for fake news detection

Multi-Level Multi-Modal Cross-Attention Network for Fake News Detection

Mutual information maximization and feature space separation and bi-bimodal mo-dality fusion for multimodal sentiment analysis

Artificial intelligence-Enabled deep learning model for multimodal biometric fusion

Multi-Modality Image Fusion and Object Detection Based on Semantic Information

Image-text sentiment analysis via deep multimodal attentive fusion.

Multimodal Information Fusion Method in Emotion Recognition in the Background of Artificial Intelligence

Application of Multimodal Data Fusion Attentive Dual Residual Generative Adversarial Network in Sentiment Recognition and Sentiment Analysis

Multimodal Fake News Detection Through Intra-Modality Feature Aggregation and Inter-Modality Semantic Fusion

Research on cross-modal emotion recognition based on multi-layer semantic fusion

Two-Level Multimodal Fusion for Sentiment Analysis in Public Security

Multi-level Multi-task Representation Learning with Adaptive Fusion for Multimodal Sentiment Analysis