Abstract:In times of crisis, the prompt and precise classification of disaster-related information shared on social media platforms is crucial for effective disaster response and public safety. During such critical events, individuals use social media to communicate, sharing multimodal textual and visual content. However, due to the significant influx of unfiltered and diverse data, humanitarian organizations face challenges in leveraging this information efficiently. Existing methods for classifying disaster-related content often fail to model users' credibility, emotional context, and social interaction information, which are essential for accurate classification. To address this gap, we propose CrisisSpot, a method that utilizes a Graph-based Neural Network to capture complex relationships between textual and visual modalities, as well as Social Context Features to incorporate user-centric and content-centric information. We also introduce Inverted Dual Embedded Attention (IDEA), which captures both harmonious and contrasting patterns within the data to enhance multimodal interactions and provide richer insights. Additionally, we present TSEqD (Turkey-Syria Earthquake Dataset), a large annotated dataset for a single disaster event, containing 10,352 samples. Through extensive experiments, CrisisSpot demonstrated significant improvements, achieving an average F1-score gain of 9.45% and 5.01% compared to state-of-the-art methods on the publicly available CrisisMMD dataset and the TSEqD dataset, respectively.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problem of how to quickly and accurately classify disaster - related multi - modal information (text and image) shared on social media platforms in emergency situations. Specifically, the authors focus on: 1. **The challenge of a large amount of unfiltered data**: During disaster events, a large amount of unscreened and diverse information will emerge on social media, which makes it difficult for humanitarian organizations to effectively use this information. 2. **Lack of modeling of user credibility, emotional context and social interaction information**: Existing methods for classifying disaster content often ignore users' social background information, such as user credibility, emotional expression and social interaction, etc. These factors are crucial for improving classification accuracy. 3. **The complexity of multi - modal data fusion**: Text and image each have different complexities, and relying solely on single - modal data analysis may not be able to capture the complete context. For example, there may be problems such as informal language and spelling mistakes in the text, while the image may lack sufficient background information. To solve the above problems, the authors propose a new method named CrisisSpot, which combines graph - based neural network and attention mechanism, especially introduces the Inverted Dual Embedded Attention (IDEA) mechanism to capture the harmony and contradiction patterns in the data. In addition, CrisisSpot also integrates Social Context Features (SCF), including User Informative Score (UIS), Crisis Informative Score (CIS) and user engagement indicators, to enhance the model's ability to understand and classify disaster content. In this way, CrisisSpot can more comprehensively understand the disaster situation and help emergency responders and humanitarian organizations more effectively classify and process information on social media, thereby improving the speed and accuracy of disaster response.

A Social Context-aware Graph-based Multimodal Attentive Learning Framework for Disaster Content Classification during Emergencies

Disaster assessment from social media using multimodal deep learning

Multimodal tweet classification in disaster response systems using transformer-based bidirectional attention model

A Named Entity Recognition and Topic Modeling-based Solution for Locating and Better Assessment of Natural Disasters in Social Media

Multi-task Multimodal Learning for Disaster Situation Assessment.

Disaster Image Classification by Fusing Multimodal Social Media Data

Rapid Classification of Crisis-Related Data on Social Networks using Convolutional Neural Networks

Robust Training of Social Media Image Classification Models for Rapid Disaster Response

Deep Learning Benchmarks and Datasets for Social Media Image Classification for Disaster Response

Multi-modal deep learning framework for damage detection in social media posts

CrisisKAN: Knowledge-infused and Explainable Multimodal Attention Network for Crisis Event Classification

CrisisMMD: Multimodal Twitter Datasets from Natural Disasters

Multimodal Social Sensing for the Spatio-Temporal Evolution and Assessment of Nature Disasters

Relevancy Classification of Multimodal Social Media Streams for Emergency Services

Graph Neural Network Enhanced Language Models for Efficient Multilingual Text Classification

CrisisMatch: Semi-Supervised Few-Shot Learning for Fine-Grained Disaster Tweet Classification

Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises

A Multimodal Data Analysis Approach to Social Media during Natural Disasters

Enhancing multimodal disaster tweet classification using state-of-the-art deep learning networks

Enhanced Arabic disaster data classification using domain adaptation

“Generalization of convolutional network to domain adaptation network for classification of disaster images on twitter”