Multichannel Multimodal Emotion Analysis of Cross-Modal Feedback Interactions Based on Knowledge Graph

Shaohua Dong,Xiaochao Fan,Xinchun Ma

DOI: https://doi.org/10.1007/s11063-024-11641-w

IF: 2.565

2024-05-30

Neural Processing Letters

Abstract:Multimodal sentiment analysis is a downstream branch task of sentiment analysis with high attention at present. Previous work in multimodal sentiment analysis have focused on the representation and fusion of modalities, capturing the underlying semantic relationships between modalities by considering contextual information. While this approach is feasible for simple contextual comments, more complex comments require the integration of external knowledge to obtain more accurate sentiment information. However, incorporating external knowledge into sentiment analysis to enhance information complementarity has not been thoroughly investigated. To address this, we propose a multichannel cross-modal feedback interaction model that incorporates the knowledge graph into multimodal sentiment analysis. Our proposed model consists of two main components: the cross-modal feedback recurrent interaction module and the external knowledge module for capturing latent information. The cross-modal interaction employs a self-feedback mechanism during network training, extracting feature representations of each modality and using these representations to mask sensory inputs, allowing the model to perform feedback-based feature masking. The external knowledge graph captures potential semantic information representations in the textual data through knowledge graph embedding. Finally, a global feature fusion module is employed for multichannel multimodal information integration. On two publicly available datasets, our method demonstrates good performance in terms of accuracy and F1 scores, compared to state-of-the-art models and several baselines.

computer science, artificial intelligence

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address several key issues in multimodal sentiment analysis:

1. **Capturing Semantic Information in Complex Contexts**: Existing methods perform well in simple contextual reviews but require the integration of external knowledge to obtain more accurate sentiment information when dealing with complex reviews. Currently, research on incorporating external knowledge into sentiment analysis to enhance information complementarity is still insufficient.

2. **Complementarity of Cross-Modal Information**: A multi-channel cross-modal feedback interaction model based on knowledge graph (MMKGE) is proposed. By introducing a knowledge graph to capture potential semantic information and utilizing a cross-modal self-feedback mechanism, the quality of feature representation is improved.

3. **Model Efficiency and Lightweight Design**: By combining cross-modal information completion and external knowledge dual channels, the semantic features are enhanced while keeping the model simple, achieving a more efficient and lightweight model design.

In summary, this paper is mainly dedicated to improving the accuracy and robustness of multimodal sentiment analysis by introducing knowledge graphs and improved cross-modal interaction mechanisms, especially in capturing implicit semantic information in complex contexts.