Abstract:The widespread use of deep learning techniques for creating realistic synthetic media, commonly known as deepfakes, poses a significant threat to individuals, organizations, and society. As the malicious use of these data could lead to unpleasant situations, it is becoming crucial to distinguish between authentic and fake media. Nonetheless, though deepfake generation systems can create convincing images and audio, they may struggle to maintain consistency across different data modalities, such as producing a realistic video sequence where both visual frames and speech are fake and consistent one with the other. Moreover, these systems may not accurately reproduce semantic and timely accurate aspects. All these elements can be exploited to perform a robust detection of fake content. In this paper, we propose a novel approach for detecting deepfake video sequences by leveraging data multimodality. Our method extracts audio-visual features from the input video over time and analyzes them using time-aware neural networks. We exploit both the video and audio modalities to leverage the inconsistencies between and within them, enhancing the final detection performance. The peculiarity of the proposed method is that we never train on multimodal deepfake data, but on disjoint monomodal datasets which contain visual-only or audio-only deepfakes. This frees us from leveraging multimodal datasets during training, which is desirable given their lack in the literature. Moreover, at test time, it allows to evaluate the robustness of our proposed detector on unseen multimodal deepfakes. We test different fusion techniques between data modalities and investigate which one leads to more robust predictions by the developed detectors. Our results indicate that a multimodal approach is more effective than a monomodal one, even if trained on disjoint monomodal datasets.

A defensive attention mechanism to detect deepfake content across multiple modalities

Multi-attentional Deepfake Detection

Refining Localized Attention Features with Multi-Scale Relationships for Enhanced Deepfake Detection in Spatial-Frequency Domain

A defensive framework for deepfake detection under adversarial settings using temporal and spatial features

Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization

A Multimodal Framework for Deepfake Detection

An Audio-Visual Attention Based Multimodal Network for Fake Talking Face Videos Detection

Detection of Deepfake Videos Using Long-Distance Attention

Multimodal Deepfake Detection for Short Videos

A Robust Approach to Multimodal Deepfake Detection

Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors

AVoiD-DF: Audio-Visual Joint Learning for Detecting Deepfake

A Unified Framework for Modality-Agnostic Deepfakes Detection

Multimodal Deepfake Detection

Audio-Visual Temporal Forgery Detection Using Embedding-Level Fusion and Multi-Dimensional Contrastive Loss

Combating deepfakes: a comprehensive multilayer deepfake video detection framework

Video Detection Method Based on Temporal and Spatial Foundations for Accurate Verification of Authenticity

Double-Stream Segmentation Network with Temporal Self-attention for Deepfake Video Detection

Deep fake detection using an optimal deep learning model with multi head attention-based feature extraction scheme

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

D-Fence layer: an ensemble framework for comprehensive deepfake detection