Abstract:As AI-generated content (AIGC) thrives, deepfakes have expanded from single-modality falsification to cross-modal fake content creation, where either audio or visual components can be manipulated. While using two unimodal detectors can detect audio-visual deepfakes, cross-modal forgery clues could be overlooked. Existing multimodal deepfake detection methods typically establish correspondence between the audio and visual modalities for binary real/fake classification, and require the co-occurrence of both modalities. However, in real-world multi-modal applications, missing modality scenarios may occur where either modality is unavailable. In such cases, audio-visual detection methods are less practical than two independent unimodal methods. Consequently, the detector can not always obtain the number or type of manipulated modalities beforehand, necessitating a fake-modality-agnostic audio-visual detector. In this work, we introduce a comprehensive framework that is agnostic to fake modalities, which facilitates the identification of multimodal deepfakes and handles situations with missing modalities, regardless of the manipulations embedded in audio, video, or even cross-modal forms. To enhance the modeling of cross-modal forgery clues, we employ audio-visual speech recognition (AVSR) as a preliminary task. This efficiently extracts speech correlations across modalities, a feature challenging for deepfakes to replicate. Additionally, we propose a dual-label detection approach that follows the structure of AVSR to support the independent detection of each modality. Extensive experiments on three audio-visual datasets show that our scheme outperforms state-of-the-art detection methods with promising performance on modality-agnostic audio/video deepfakes.

Appearance Matters, So Does Audio: Revealing the Hidden Face via Cross-Modality Transfer

APB2FaceV2: Real-Time Audio-Guided Multi-Face Reenactment

APB2FACE: Audio-Guided Face Reenactment with Auxiliary Pose and Blink Signals.

Ghost-in-Wave: How Speaker-Irrelative Features Interfere DeepFake Voice Detectors

Real-Time Audio-Guided Multi-Face Reenactment

Transferring Audio Deepfake Detection Capability Across Languages

Audio-Driven Identity Manipulation for Face Inpainting

Restore DeepFakes Video Frames Via Identifying Individual Motion Styles

FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio

An Audio-Visual Attention Based Multimodal Network for Fake Talking Face Videos Detection

AVForensics: Audio-driven Deepfake Video Detection with Masking Strategy in Self-supervision.

Voice-Face Homogeneity Tells Deepfake

RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network

Self-Attention and Hybrid Features for Replay and Deep-Fake Audio Detection

Parametric Implicit Face Representation for Audio-Driven Facial Reenactment

Deep Face Swapping via Cross-Identity Adversarial Training.

Audio-Visual Contrastive Pre-train for Face Forgery Detection

Study of detecting behavioral signatures within DeepFake videos

SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Space

A Unified Framework for Modality-Agnostic Deepfakes Detection

MFAAN: Unveiling Audio Deepfakes with a Multi-Feature Authenticity Network