Abstract:Multimodal Sentiment Analysis (SA) is gaining popularity due to its broad application potential. The existing studies have focused on the SA of single modalities, such as texts or photos, posing challenges in effectively handling social media data with multiple modalities. Moreover, most multimodal research has concentrated on merely combining the two modalities rather than exploring their complex correlations, leading to unsatisfactory sentiment classification results. Motivated by this, we propose a new visual-textual sentiment classification model named Multi-Model Fusion (MMF), which uses a mixed fusion framework for SA to effectively capture the essential information and the intrinsic relationship between the visual and textual content. The proposed model comprises three deep neural networks. Two different neural networks are proposed to extract the most emotionally relevant aspects of image and text data. Thus, more discriminative features are gathered for accurate sentiment classification. Then, a multichannel joint fusion model with a self-attention technique is proposed to exploit the intrinsic correlation between visual and textual characteristics and obtain emotionally rich information for joint sentiment classification. Finally, the results of the three classifiers are integrated using a decision fusion scheme to improve the robustness and generalizability of the proposed model. An interpretable visual-textual sentiment classification model is further developed using the Local Interpretable Model-agnostic Explanation model (LIME) to ensure the model’s explainability and resilience. The proposed MMF model has been tested on four real-world sentiment datasets, achieving (99.78%) accuracy on Binary_Getty (BG), (99.12%) on Binary_iStock (BIS), (95.70%) on Twitter, and (79.06%) on the Multi-View Sentiment Analysis (MVSA) dataset. These results demonstrate the superior performance of our MMF model compared to single-model approaches and current state-of-the-art techniques based on model evaluation criteria.

Multi-modal Feature Fistillation Emotion Recognition Method for Social Media

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Sentiment Analysis Based on Text Information Enhancement and Multimodal Feature Fusion

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Research on Image-text Multimodal Emotions Analysis with Fused Emoji

MIECF: Multi-faceted information extraction and cross-mixture fusion for multimodal aspect-based sentiment analysis

Exploiting Visual Context and Multi-grained Semantics for Social Text Emotion Recognition

FV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition Inference

Image-text sentiment analysis via deep multimodal attentive fusion.

A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

Multimodal Sentiment Analysis of Social Media Based on Top-Layer Fusion

EMFSA: Emoji-based multifeature fusion sentiment analysis

Multi-granularity Visual-Textual Jointly Modeling for Aspect-Level Multimodal Sentiment Analysis

Design and Efficacy of a Data Lake Architecture for Multimodal Emotion Feature Extraction in Social Media

Extracting Method for Fine-Grained Emotional Features in Videos

A multimodal sentiment recognition method based on attention mechanism

Social Media Public Opinion Detection Using Multimodal Natural Language Processing and Attention Mechanisms

Multi-Model Fusion Framework Using Deep Learning for Visual-Textual Sentiment Classification

Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism