Abstract:In multimodal sentiment analysis, achieving effective fusion among text, acoustic, and visual modalities for enhanced sentiment prediction is a crucial research topic. Recent studies typically employ tensor-based or attention-based mechanisms for multimodal fusion. However, the former fails to achieve satisfactory prediction performance, and the latter complicates the computation of fusion between non-textual modalities. Therefore, this paper proposes the multimodal sentiment analysis model based on Adaptive Feature Normalization and Attention Gating mechanism (AdaFN-AG). Firstly, facing highly synchronized non-textual modalities, we design the Adaptive Feature Normalization (AdaFN) method, which focuses more on sentiment features interaction rather than timing features association. In AdaFN, acoustic and visual modality features achieve cross-modal interaction through normalization, inverse normalization, and mix-up operations, with weights utilized for adaptive strength regulation of the cross-modal interaction. Meanwhile, we design the Attention Gating mechanism that facilitates cross-modal interactions between textual and non-textual modalities through cross-attention and captures timing associations, while the gating module concurrently regulates the intensity of these interactions. Additionally, we employ self-attention to capture the intrinsic correlations within single-modal features. Subsequently, we conduct experiments on three benchmark datasets for multimodal sentiment analysis, with the results indicating that AdaFN-AG outperforms the baselines across the majority of evaluation metrics. Through research and experiments, we validate that AdaFN-AG not only enhances performance by adopting appropriate methods for different types of cross-modal interactions while conserving computational resources but also verifies the generalization capability of the AdaFN method.

AdaMoW: Multimodal Sentiment Analysis Based on Adaptive Modality-Specific Weight Fusion Network

Heterogeneous Hierarchical Fusion Network for Multimodal Sentiment Analysis in Real-World Environments

Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Multimodal Multi-loss Fusion Network for Sentiment Analysis

AMSA: Adaptive Multimodal Learning for Sentiment Analysis

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

MSAF: Multimodal Split Attention Fusion

Multimodal Sentiment Analysis Method Based on Hierarchical Adaptive Feature Fusion Network

Adafn-Ag: Enhancing Multimodal Interaction with Adaptive Feature Normalization for Multimodal Sentiment Analysis

Multi-level Attention Map Network for Multimodal Sentiment Analysis

Mutual information maximization and feature space separation and bi-bimodal mo-dality fusion for multimodal sentiment analysis

TMMDA: A New Token Mixup Multimodal Data Augmentation for Multimodal Sentiment Analysis

TeFNA: Text-centered Fusion Network with crossmodal Attention for multimodal sentiment analysis

Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks

CSMF-SPC: Multimodal Sentiment Analysis Model with Effective Context Semantic Modality Fusion and Sentiment Polarity Correction

Dynamic Weighted Multitask Learning and Contrastive Learning for Multimodal Sentiment Analysis

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention

A text guided multi-task learning network for multimodal sentiment analysis