Abstract:Abstract Purpose Nowadays, public opinions during public emergencies involve not only textual contents but also contain images. However, the existing works mainly focus on textual contents and they do not provide a satisfactory accuracy of sentiment analysis, lacking the combination of multimodal contents. In this paper, we propose to combine texts and images generated in the social media to perform sentiment analysis. Design/methodology/approach We propose a Deep Multimodal Fusion Model (DMFM), which combines textual and visual sentiment analysis. We first train word2vec model on a large-scale public emergency corpus to obtain semantic-rich word vectors as the input of textual sentiment analysis. BiLSTM is employed to generate encoded textual embeddings. To fully excavate visual information from images, a modified pretrained VGG16-based sentiment analysis network is used with the best-performed fine-tuning strategy. A multimodal fusion method is implemented to fuse textual and visual embeddings completely, producing predicted labels. Findings We performed extensive experiments on Weibo and Twitter public emergency datasets, to evaluate the performance of our proposed model. Experimental results demonstrate that the DMFM provides higher accuracy compared with baseline models. The introduction of images can boost the performance of sentiment analysis during public emergencies. Research limitations In the future, we will test our model in a wider dataset. We will also consider a better way to learn the multimodal fusion information. Practical implications We build an efficient multimodal sentiment analysis model for the social media contents during public emergencies. Originality/value We consider the images posted by online users during public emergencies on social platforms. The proposed method can present a novel scope for sentiment analysis during public emergencies and provide the decision support for the government when formulating policies in public emergencies.

A Multimodal Resource Balancing Approach Based on Cross-Attention and Bidrectional Associative Memory Networks

Balanced sentimental information via multimodal interaction model

Multi-task Multimodal Learning for Disaster Situation Assessment.

Multimodal Deep Learning Based on Multiple Correspondence Analysis for Disaster Management

A Multimodal Data Analysis Approach to Social Media during Natural Disasters

Multimodal sentiment analysis based on multiple attention

Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

Multimodal Sentiment Analysis Missing Modality Reconstruction Network Based on Shared-Specific Features

Multimodal sentiment analysis for social media contents during public emergencies

Modality-Specific Cross-Modal Similarity Measurement With Recurrent Attention Network

Mutual information maximization and feature space separation and bi-bimodal mo-dality fusion for multimodal sentiment analysis

Multimodal Fusion Balancing Through Game-Theoretic Regularization

Multimodal Sentiment Analysis Based on Information Bottleneck and Attention Mechanisms

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Exploring Multimodal Sentiment Analysis via CBAM Attention and Double-layer BiLSTM Architecture

Generating and encouraging: An effective framework for solving class imbalance in multimodal emotion recognition conversation

Multimodal Social Sensing for the Spatio-Temporal Evolution and Assessment of Nature Disasters

Disaster Image Classification by Fusing Multimodal Social Media Data

An efficient multimodal sentiment analysis in social media using hybrid optimal multi-scale residual attention network

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

Heterogeneous Hierarchical Fusion Network for Multimodal Sentiment Analysis in Real-World Environments