Abstract:The widespread use of deep learning techniques for creating realistic synthetic media, commonly known as deepfakes, poses a significant threat to individuals, organizations, and society. As the malicious use of these data could lead to unpleasant situations, it is becoming crucial to distinguish between authentic and fake media. Nonetheless, though deepfake generation systems can create convincing images and audio, they may struggle to maintain consistency across different data modalities, such as producing a realistic video sequence where both visual frames and speech are fake and consistent one with the other. Moreover, these systems may not accurately reproduce semantic and timely accurate aspects. All these elements can be exploited to perform a robust detection of fake content. In this paper, we propose a novel approach for detecting deepfake video sequences by leveraging data multimodality. Our method extracts audio-visual features from the input video over time and analyzes them using time-aware neural networks. We exploit both the video and audio modalities to leverage the inconsistencies between and within them, enhancing the final detection performance. The peculiarity of the proposed method is that we never train on multimodal deepfake data, but on disjoint monomodal datasets which contain visual-only or audio-only deepfakes. This frees us from leveraging multimodal datasets during training, which is desirable given their lack in the literature. Moreover, at test time, it allows to evaluate the robustness of our proposed detector on unseen multimodal deepfakes. We test different fusion techniques between data modalities and investigate which one leads to more robust predictions by the developed detectors. Our results indicate that a multimodal approach is more effective than a monomodal one, even if trained on disjoint monomodal datasets.

Multi-modal Semantic Inconsistency Detection in Social Media News Posts

Misinformation Detection in Social Media Video Posts

Semantic‐enhanced multimodal fusion network for fake news detection

Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approach

Look and Listen: A Multi-modality Late Fusion Approach to Scene Classification for Autonomous Machines

Human Cognition-based Consistency Inference Networks for Multi-modal Fake News Detection

VMID: A Multimodal Fusion LLM Framework for Detecting and Identifying Misinformation of Short Videos

Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach

A Robust Approach to Multimodal Deepfake Detection

Adaptive Learning of Consistency and Inconsistency Information for Fake News Detection

Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion

A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection

Exploring the Role of Audio in Multimodal Misinformation Detection

Unsupervised Multimodal Deepfake Detection Using Intra- and Cross-Modal Inconsistencies

An image and text-based multimodal model for detecting fake news in OSN's

Multimodal Deepfake Detection for Short Videos

Multimodal Classification for Analysing Social Media

Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection

Cross-modal Consistency Learning with Fine-grained Fusion Network for Multimodal Fake News Detection

Text-image multimodal fusion model for enhanced fake news detection

MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection