Abstract:The rapid evolution of the digital era has greatly transformed social media, resulting in more diverse emotional expressions and increasingly complex public discourse. Consequently, identifying relationships within multimodal data has become increasingly challenging. Most current multimodal sentiment analysis (MSA) methods concentrate on merging data from diverse modalities into an integrated feature representation to enhance recognition performance by leveraging the complementary nature of multimodal data. However, these approaches often overlook prediction reliability. To address this, we propose the uncertainty estimation fusion network (UEFN), a reliable MSA method based on uncertainty estimation. UEFN combines the Dirichlet distribution and Dempster-Shafer evidence theory (DSET) to predict the probability distribution and uncertainty of text, speech, and image modalities, fusing the predictions at the decision level. Specifically, the method first represents the contextual features of text, speech, and image modalities separately. It then employs a fully connected neural network to transform features from different modalities into evidence forms. Subsequently, it parameterizes the evidence of different modalities via the Dirichlet distribution and estimates the probability distribution and uncertainty for each modality. Finally, we use DSET to fuse the predictions, obtaining the sentiment analysis results and uncertainty estimation, referred to as the multimodal decision fusion layer (MDFL). Additionally, on the basis of the modality uncertainty generated by subjective logic theory, we calculate feature weights, apply them to the corresponding features, concatenate the weighted features, and feed them into a feedforward neural network for sentiment classification, forming the adaptive weight fusion layer (AWFL). Both MDFL and AWFL are then used for multitask training. Experimental comparisons demonstrate that the UEFN not only achieves excellent performance but also provides uncertainty estimation along with the predictions, enhancing the reliability and interpretability of the results.

Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion

Uncertainty-Debiased Multimodal Fusion: Learning Deterministic Joint Representation for Multimodal Sentiment Analysis

Enhancing Unimodal Features Matters: A Multimodal Framework for Building Extraction.

Hybrid Uncertainty Calibration for Multimodal Sentiment Analysis

MEFusion: Unsupervised Mutual Enhancement for Multimodal Image Fusion

UEFN: Efficient uncertainty estimation fusion network for reliable multimodal sentiment analysis

Sparsely Multimodal Data Fusion

Enhancing Multimodal Fusion with Only Unimodal Data

Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations

Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences

Context-aware Multimodal Fusion for Emotion Recognition

MSAF: Multimodal Split Attention Fusion

Modeling Both Intra- and Inter-Modality Uncertainty for Multimodal Fake News Detection

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

Mutual information maximization and feature space separation and bi-bimodal mo-dality fusion for multimodal sentiment analysis

Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes

Learning to Learn Better Unimodal Representations Via Adaptive Multimodal Meta-Learning

COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion

Weakly paired multimodal fusion using multilayer extreme learning machine