Abstract:The rapid evolution of the digital era has greatly transformed social media, resulting in more diverse emotional expressions and increasingly complex public discourse. Consequently, identifying relationships within multimodal data has become increasingly challenging. Most current multimodal sentiment analysis (MSA) methods concentrate on merging data from diverse modalities into an integrated feature representation to enhance recognition performance by leveraging the complementary nature of multimodal data. However, these approaches often overlook prediction reliability. To address this, we propose the uncertainty estimation fusion network (UEFN), a reliable MSA method based on uncertainty estimation. UEFN combines the Dirichlet distribution and Dempster-Shafer evidence theory (DSET) to predict the probability distribution and uncertainty of text, speech, and image modalities, fusing the predictions at the decision level. Specifically, the method first represents the contextual features of text, speech, and image modalities separately. It then employs a fully connected neural network to transform features from different modalities into evidence forms. Subsequently, it parameterizes the evidence of different modalities via the Dirichlet distribution and estimates the probability distribution and uncertainty for each modality. Finally, we use DSET to fuse the predictions, obtaining the sentiment analysis results and uncertainty estimation, referred to as the multimodal decision fusion layer (MDFL). Additionally, on the basis of the modality uncertainty generated by subjective logic theory, we calculate feature weights, apply them to the corresponding features, concatenate the weighted features, and feed them into a feedforward neural network for sentiment classification, forming the adaptive weight fusion layer (AWFL). Both MDFL and AWFL are then used for multitask training. Experimental comparisons demonstrate that the UEFN not only achieves excellent performance but also provides uncertainty estimation along with the predictions, enhancing the reliability and interpretability of the results.

EFUI: An ensemble framework using uncertain inference for pornographic image recognition.

Sensitive Information Detection Based on Deep Learning Models

FEUI: Fusion Embedding for User Identification Across Social Networks

Fine-grained Pornographic Image Recognition with Multiple Feature Fusion Transfer Learning

Intuitionistic fuzzy theory-neural network basing pornographic image recognizing algorithm

A Pornographic Images Recognition Model Based on Deep One-Class Classification with Visual Attention Mechanism.

Adult Image and Video Recognition by a Deep Multicontext Network and Fine-to-Coarse Strategy.

Multilevel fusion of multimodal deep features for porn streamer recognition in live video

Pornographic Video Detection Based on Semantic and Image Enhancement

UEFN: Efficient uncertainty estimation fusion network for reliable multimodal sentiment analysis

Deep Evidential Fusion Network for Image Classification

Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision Strategy for Facial Expression Recognition in the Wild

Uncertainty Inspired Underwater Image Enhancement

DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference in Image Recognition.

Pornographic Image Recognition via Weighted Multiple Instance Learning

Automated Nudity Recognition using Very Deep Residual Learning Network

A Novel Scheme for Intelligent Recognition of Pornographic Images

Enhancing Facial Expression Recognition Under Data Uncertainty Based on Embedding Proximity

UFCC: A Unified Forensic Approach to Locating Tampered Areas in Still Images and Detecting Deepfake Videos by Evaluating Content Consistency

FCNet: A Feature Context Network Based on Ensemble Framework for Image Retrieval

DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference