Abstract:Multimodal Sentiment Analysis (SA) is gaining popularity due to its broad application potential. The existing studies have focused on the SA of single modalities, such as texts or photos, posing challenges in effectively handling social media data with multiple modalities. Moreover, most multimodal research has concentrated on merely combining the two modalities rather than exploring their complex correlations, leading to unsatisfactory sentiment classification results. Motivated by this, we propose a new visual-textual sentiment classification model named Multi-Model Fusion (MMF), which uses a mixed fusion framework for SA to effectively capture the essential information and the intrinsic relationship between the visual and textual content. The proposed model comprises three deep neural networks. Two different neural networks are proposed to extract the most emotionally relevant aspects of image and text data. Thus, more discriminative features are gathered for accurate sentiment classification. Then, a multichannel joint fusion model with a self-attention technique is proposed to exploit the intrinsic correlation between visual and textual characteristics and obtain emotionally rich information for joint sentiment classification. Finally, the results of the three classifiers are integrated using a decision fusion scheme to improve the robustness and generalizability of the proposed model. An interpretable visual-textual sentiment classification model is further developed using the Local Interpretable Model-agnostic Explanation model (LIME) to ensure the model’s explainability and resilience. The proposed MMF model has been tested on four real-world sentiment datasets, achieving (99.78%) accuracy on Binary_Getty (BG), (99.12%) on Binary_iStock (BIS), (95.70%) on Twitter, and (79.06%) on the Multi-View Sentiment Analysis (MVSA) dataset. These results demonstrate the superior performance of our MMF model compared to single-model approaches and current state-of-the-art techniques based on model evaluation criteria.

MSMF: Multi-Scale Multi-Modal Fusion for Enhanced Stock Market Prediction

Stock Trend Prediction with Multi-Granularity Data: A Contrastive Learning Approach with Adaptive Fusion

Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

A model fusion method based on multi-source heterogeneous data for stock trading signal prediction

MF-DAT: a stock trend prediction of the double-graph attention network based on multisource information fusion

Background-aware Multi-source Fusion Financial Trend Forecasting Mechanism

Learning To Fuse Multiple Semantic Aspects From Rich Texts For Stock Price Prediction

Multi-scale contrast approach for stock index prediction with adaptive stock fusion

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

Multi-Model Fusion Method and its Application in Prediction of Stock Index Movements

Multimodal multiscale dynamic graph convolution networks for stock price prediction

A Stock Prediction Method Based on Multidimensional and Multilevel Feature Dynamic Fusion

Multi-modal Attention Network for Stock Movements Prediction

Residual long short-term memory network with multi-source and multi-frequency information fusion: An application to China's stock market

Multi-modal transform-based fusion model for new product sales forecasting

Multi-feature fusion stock prediction based on knowledge graph

MSAF: Multimodal Split Attention Fusion

Multi-Model Fusion Framework Using Deep Learning for Visual-Textual Sentiment Classification

M$^{3}$SA: Multimodal Sentiment Analysis Based on Multi-Scale Feature Extraction and Multi-Task Learning

Residual long short-term memory network with multi-source and multi-frequency information fusion: An application to China?s stock market