Abstract:The widespread use of deep learning techniques for creating realistic synthetic media, commonly known as deepfakes, poses a significant threat to individuals, organizations, and society. As the malicious use of these data could lead to unpleasant situations, it is becoming crucial to distinguish between authentic and fake media. Nonetheless, though deepfake generation systems can create convincing images and audio, they may struggle to maintain consistency across different data modalities, such as producing a realistic video sequence where both visual frames and speech are fake and consistent one with the other. Moreover, these systems may not accurately reproduce semantic and timely accurate aspects. All these elements can be exploited to perform a robust detection of fake content. In this paper, we propose a novel approach for detecting deepfake video sequences by leveraging data multimodality. Our method extracts audio-visual features from the input video over time and analyzes them using time-aware neural networks. We exploit both the video and audio modalities to leverage the inconsistencies between and within them, enhancing the final detection performance. The peculiarity of the proposed method is that we never train on multimodal deepfake data, but on disjoint monomodal datasets which contain visual-only or audio-only deepfakes. This frees us from leveraging multimodal datasets during training, which is desirable given their lack in the literature. Moreover, at test time, it allows to evaluate the robustness of our proposed detector on unseen multimodal deepfakes. We test different fusion techniques between data modalities and investigate which one leads to more robust predictions by the developed detectors. Our results indicate that a multimodal approach is more effective than a monomodal one, even if trained on disjoint monomodal datasets.

Multimodal Deepfake Detection

A Multimodal Framework for Deepfake Detection

Integrating Audio-Visual Features for Multimodal Deepfake Detection

A Robust Approach to Multimodal Deepfake Detection

Multimodal Deepfake Detection for Short Videos

Multimodaltrace: Deepfake Detection using Audiovisual Representation Learning

Combating deepfakes: a comprehensive multilayer deepfake video detection framework

MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection

Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors

Deepfake Detection System Using Deep Neural Networks

Deep fake video/image detection using deep learning

Experimental Results on Multi-modal Deepfake Detection

A Unified Framework for Modality-Agnostic Deepfakes Detection

Unsupervised Multimodal Deepfake Detection Using Intra- and Cross-Modal Inconsistencies

A defensive attention mechanism to detect deepfake content across multiple modalities

Understanding Audiovisual Deepfake Detection: Techniques, Challenges, Human Factors and Perceptual Insights

Deepfake Detection Based on Temporal Analysis of Facial Dynamics Using LSTM and ResNeXt Architectures

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Synthetic Content Detection in Deepfake Video using Deep Learning

Deep Fake Detection: Survey of Facial Manipulation Detection Solutions

Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization