Abstract:Multimodal Sentiment Analysis (SA) is gaining popularity due to its broad application potential. The existing studies have focused on the SA of single modalities, such as texts or photos, posing challenges in effectively handling social media data with multiple modalities. Moreover, most multimodal research has concentrated on merely combining the two modalities rather than exploring their complex correlations, leading to unsatisfactory sentiment classification results. Motivated by this, we propose a new visual-textual sentiment classification model named Multi-Model Fusion (MMF), which uses a mixed fusion framework for SA to effectively capture the essential information and the intrinsic relationship between the visual and textual content. The proposed model comprises three deep neural networks. Two different neural networks are proposed to extract the most emotionally relevant aspects of image and text data. Thus, more discriminative features are gathered for accurate sentiment classification. Then, a multichannel joint fusion model with a self-attention technique is proposed to exploit the intrinsic correlation between visual and textual characteristics and obtain emotionally rich information for joint sentiment classification. Finally, the results of the three classifiers are integrated using a decision fusion scheme to improve the robustness and generalizability of the proposed model. An interpretable visual-textual sentiment classification model is further developed using the Local Interpretable Model-agnostic Explanation model (LIME) to ensure the model’s explainability and resilience. The proposed MMF model has been tested on four real-world sentiment datasets, achieving (99.78%) accuracy on Binary_Getty (BG), (99.12%) on Binary_iStock (BIS), (95.70%) on Twitter, and (79.06%) on the Multi-View Sentiment Analysis (MVSA) dataset. These results demonstrate the superior performance of our MMF model compared to single-model approaches and current state-of-the-art techniques based on model evaluation criteria.

Multi-level fusion with deep neural networks for multimodal sentiment classification

Multi-level Multi-task Representation Learning with Adaptive Fusion for Multimodal Sentiment Analysis

Multimodal Multi-loss Fusion Network for Sentiment Analysis

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

Multi-Channel Attentive Graph Convolutional Network with Sentiment Fusion for Multimodal Sentiment Analysis

Multi‐level Deep Correlative Networks for Multi‐modal Sentiment Analysis

Tri-Modalities Fusion for Multimodal Sentiment Analysis

MFSC: A Multimodal Aspect-Level Sentiment Classification Framework with Multi-Image Gate and Fusion Networks

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Two-Level Multimodal Fusion for Sentiment Analysis in Public Security

Multi-Feature Fusion for Multimodal Attentive Sentiment Analysis.

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Exploring Multimodal Multiscale Features for Sentiment Analysis Using Fuzzy-Deep Neural Network Learning

Multi-Model Fusion Framework Using Deep Learning for Visual-Textual Sentiment Classification

Dense Fusion Network with Multimodal Residual for Sentiment Classification.

Image Sentiment Analysis Method Based on Multi-Level Feature Fusion

Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks

A Multi-model Fusion Framework Based on Deep Learning for Sentiment Classification.

Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature Fusion

Multimodal Sentiment Analysis Based on Composite Hierarchical Fusion