Abstract:Multimodal sentiment analysis faces two challenges: modality representation and modality fusion. Most of the existing models rely only on the feature extraction network to learn modality representation, and the fusion mechanism adopted by some models does not perform well. These factors are not conducive to the model learning rich emotional information and further affect the model’s predictive ability. To solve these problems, we propose a multimodal sentiment analysis model based on two-stage contrastive learning and feature hierarchical fusion network (TSCL-FHFN). First, we apply the idea of contrastive learning to unimodal feature representation and multimodal fusion feature representation respectively. By designing a two-stage contrastive learning task, TSCL-FHFN learns similar features for data with the same emotion category and learns distinguishable features for data with different emotion categories. This enables the model to better learn the features of emotional differences. Second, in order to further explore the deep semantic association of multimodal data, we propose a multimodal feature hierarchical fusion network (FHFN). The core idea is to design an attention-based directional cross-modal transformer so that one modality can receive information from the other modality, thereby obtaining complementary information between two modalities. Then, FHFN uses the low-rank tensor fusion method to further learn interactive information between multiple modalities. Finally, we conduct a series of comparative experiments on CMU-MOSI and CMU-MOSEI datasets. Compared with the current representative models, the TSCL-FHFN model achieves better experimental results. In addition, ablation experiments further verify the effectiveness of the improved TSCL-FHFN model.

A Graph Convolution-Based Heterogeneous Fusion Network for Multimodal Sentiment Analysis

Multi-Channel Attentive Graph Convolutional Network with Sentiment Fusion for Multimodal Sentiment Analysis

Heterogeneous Hierarchical Fusion Network for Multimodal Sentiment Analysis in Real-World Environments

Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks

NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

CTHFNet: contrastive translation and hierarchical fusion network for text–video–audio sentiment analysis

DGFN Multimodal Emotion Analysis Model Based on Dynamic Graph Fusion Network

Tri-Modalities Fusion for Multimodal Sentiment Analysis

Multimodal sentiment analysis based on cross-instance graph neural networks

Multimodal Multi-loss Fusion Network for Sentiment Analysis

A transformer-encoder-based multimodal multi-attention fusion network for sentiment analysis

Tensor Fusion Network for Multimodal Sentiment Analysis

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

Multimodal Sentiment Analysis Based on Composite Hierarchical Fusion

TSCL-FHFN: two-stage contrastive learning and feature hierarchical fusion network for multimodal sentiment analysis

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

A Relation-Aware Heterogeneous Graph Transformer on Dynamic Fusion for Multimodal Classification Tasks

Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognition

Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion

Multimodal Sentiment Analysis Representations Learning via Contrastive Learning with Condense Attention Fusion