Abstract:Multimodal hateful social media meme detection is an important and challenging problem in the vision-language domain. Recent studies show high accuracy for such multimodal tasks due to datasets that provide better joint multimodal embedding to narrow the semantic gap. Religiously hateful meme detection is not extensively explored among published datasets. While there is a need for higher accuracy on religiously hateful memes, deep learning-based models often suffer from inductive bias. This issue is addressed in this work with the following contributions. First, a religiously hateful memes dataset is created and published publicly to advance hateful religious memes detection research. Over 2000 meme images are collected with their corresponding text. The proposed approach compares and fine-tunes VisualBERT pre-trained on the Conceptual Caption (CC) dataset for the downstream classification task. We also extend the dataset with the Facebook hateful memes dataset. We extract visual features using ResNeXT-152 Aggregated Residual Transformations based Masked Regions with Convolutional Neural Networks (R-CNN) and Bidirectional Encoder Representations from Transformers (BERT) uncased for textual encoding for the early fusion model. We use the primary evaluation metric of an Area Under the Operator Characters Curve (AUROC) to measure model separability. Results show that the proposed approach has a higher AUCROC score of \(78\% \) , proving the model’s higher separability performance and an accuracy of \(70\% \) . It shows comparatively superior performance considering dataset size and against ensemble-based machine learning approaches.

The registry of the European Working Group on Cardiac Pacing (EWGCP). A working group of the European Society of Cardiology.

Hateful Memes Detection via Complementary Visual and Linguistic Networks

A Multimodal Framework for the Detection of Hateful Memes

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning

Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge

Two-Way Feature Extraction Using Sequential and Multimodal Approach for Hateful Meme Classification

Feels Bad Man: Dissecting Automated Hateful Meme Detection Through the Lens of Facebook's Challenge

OSPC: Detecting Harmful Memes with Large Language Model as a Catalyst

Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models

MemeFier: Dual-stage Modality Fusion for Image Meme Classification

Multimodal Deep Learning with Discriminant Descriptors for Offensive Memes Detection

HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes

OSPC: Artificial VLM Features for Hateful Meme Detection

On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learning

Multimodal Religiously Hateful Social Media Memes Classification based on Textual and Image Data

Meme Sentiment Analysis Enhanced with Multimodal Spatial Encoding and Facial Embedding

Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection

MHS-STMA: Multimodal Hate Speech Detection via Scalable Transformer-Based Multilevel Attention Framework