Abstract:As one of the most popular social media platforms, microblogs are ideal places for news propagation. In microblogs, tweets with both text and images are more likely to attract attention than text-only tweets. This advantage is exploited by fake news producers to publish fake news, which has a devasting impact on individuals and society. Thus, multimodal fake news detection has attracted the attention of many researchers. For news with text and image, multimodal fake news detection utilizes both text and image information to determine the authenticity of news. Most of the existing methods for multimodal fake news detection obtain a joint representation by simply concatenating a vector representation of the text and a visual representation of the image, which ignores the dependencies between them. Although there are a small number of approaches that use the attention mechanism to fuse them, they are not fine-grained enough in feature fusion. The reason is that, for a given image, there are multiple visual features and certain correlations between these features. They do not use multiple feature vectors representing different visual features to fuse with textual features, and ignore the correlations, resulting in inadequate fusion of textual features and visual features. In this paper, we propose a novel fine-grained multimodal fusion network (FMFN) to fully fuse textual features and visual features for fake news detection. Scaled dot-product attention is utilized to fuse word embeddings of words in the text and multiple feature vectors representing different features of the image, which not only considers the correlations between different visual features but also better captures the dependencies between textual features and visual features. We conduct extensive experiments on a public Weibo dataset. Our approach achieves competitive results compared with other methods for fusing visual representation and text representation, which demonstrates that the joint representation learned by the FMFN (which fuses multiple visual features and multiple textual features) is better than the joint representation obtained by fusing a visual representation and a text representation in determining fake news.

Three-way Decisions Based Feature Fusion for Chinese Irony Detection

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

Integrating GIN-based multimodal feature transformation and multi-feature combination voting for irony-aware cyberbullying detection

Emotion Detection in Online Social Networks: A Multilabel Learning Approach

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

A Multi-dimension and Multi-granularity Feature Fusion Method for Chinese Microblog Sentiment Classification

A Neural Approach to Irony Generation

Stance Detection Based on User Feature Fusion

Multi-feature fusion framework for sarcasm identification on twitter data: A machine learning based approach

False Information Detection via Multimodal Feature Fusion and Multi-Classifier Hybrid Prediction

An Attention-Based Multi-Representational Fusion Method for Social-Media-Based Text Classification

Augmenting emotion features in irony detection with Large language modeling

BCMF: A bidirectional cross-modal fusion model for fake news detection

Detect Sarcasm and Humor Jointly by Neural Multi-Task Learning

A Sentiment Classification Method of Web Social Media Based on Multidimensional and Multilevel Modeling

Deep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

FMFN: Fine-Grained Multimodal Fusion Networks for Fake News Detection

THU_NGN at SemEval-2018 Task 3: Tweet Irony Detection with Densely connected LSTM and Multi-task Learning.

Dual Co-Attention-Based Multi-Feature Fusion Method for Rumor Detection

A Relation-Aware Heterogeneous Graph Transformer on Dynamic Fusion for Multimodal Classification Tasks