Abstract:Interactive fusion methods have been successfully applied to multimodal sentiment analysis, due to their ability to achieve data complementarity via interaction of different modalities. However, previous methods treat the information of each modality as a whole and usually treat them equally, failing to distinguish the contribution of different semantic regions in non-textual features towards textual features. It caused that the public regions fail to be captured and private regions are hard to be predicted only with textual. Meanwhile, these methods use sentiment-independent encoder to encode textual features, which may mistakenly identify syntactically irrelevant contextual words as clues for predicting sentiment. In this paper, we propose a coordinated-joint translation fusion framework with sentiment-interactive graph to solve these problems. Specifically, we generate a novel sentiment-interactive graph to incorporate sentiment associations between different words into the syntactic adjacency matrix. The relationships between nodes are no longer limited to the sole existence of syntactic associations but fully consider the interaction of sentiment between different words. Then, we design a coordinated-joint translation fusion module. This module utilizes a cross-modal masked attention mechanism to determine whether there is a correlation between the text and non-text inputs, thereby identifying the most relevant public semantic features in the visual and acoustic modalities corresponding to the text modality. Subsequently, a cross-modal translation-aware mechanism is used to calculate the differences between the visual and acoustic modalities features transformed into the text modality and the text modality itself, which allows us to reconstruct the visual and acoustic modalities towards text modality to obtain private semantic features. In addition, we construct a multimodal fusion layer to fuse textual features and non-textual public and private features to improve multimodal interaction effects. Experimental results on publicly available datasets CMU-MOSI and CMU-MOSEI illustrate that our proposed model achieve a best accuracy of 86.5% and 86.1%, and best F1 of 86.4% and 86.1%. A series of further analyses also indicate the proposed framework effectively improve the sentiment identification capability.

NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis

CTHFNet: contrastive translation and hierarchical fusion network for text–video–audio sentiment analysis

Heterogeneous Hierarchical Fusion Network for Multimodal Sentiment Analysis in Real-World Environments

A Graph Convolution-Based Heterogeneous Fusion Network for Multimodal Sentiment Analysis

TeFNA: Text-centered Fusion Network with crossmodal Attention for multimodal sentiment analysis

Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks

NRAFN: a non-text reinforcement and adaptive fusion network for multimodal sentiment analysis

A Cross-Modal Fusion Network Based on Self-Attention and Residual Structure for Multimodal Emotion Recognition

Sentiment Analysis of Social Media Comments Based on Multimodal Attention Fusion Network

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature Fusion

SKEAFN: Sentiment Knowledge Enhanced Attention Fusion Network for multimodal sentiment analysis

Text-centered Cross-Sample Fusion Network for Multimodal Sentiment Analysis

Interactive Fusion Network with Recurrent Attention for Multimodal Aspect-based Sentiment Analysis.

Tri-Modalities Fusion for Multimodal Sentiment Analysis

Dynamically Adjust Word Representations Using Unaligned Multimodal Information

Multimodal Multi-loss Fusion Network for Sentiment Analysis

Adafn-Ag: Enhancing Multimodal Interaction with Adaptive Feature Normalization for Multimodal Sentiment Analysis

BAFN: Bi-Direction Attention Based Fusion Network for Multimodal Sentiment Analysis

Coordinated-joint Translation Fusion Framework with Sentiment-Interactive Graph Convolutional Networks for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis Network Based on Distributional Transformation and Gated Cross-Modal Fusion