Abstract:As AI-generated content (AIGC) thrives, deepfakes have expanded from single-modality falsification to cross-modal fake content creation, where either audio or visual components can be manipulated. While using two unimodal detectors can detect audio-visual deepfakes, cross-modal forgery clues could be overlooked. Existing multimodal deepfake detection methods typically establish correspondence between the audio and visual modalities for binary real/fake classification, and require the co-occurrence of both modalities. However, in real-world multi-modal applications, missing modality scenarios may occur where either modality is unavailable. In such cases, audio-visual detection methods are less practical than two independent unimodal methods. Consequently, the detector can not always obtain the number or type of manipulated modalities beforehand, necessitating a fake-modality-agnostic audio-visual detector. In this work, we introduce a comprehensive framework that is agnostic to fake modalities, which facilitates the identification of multimodal deepfakes and handles situations with missing modalities, regardless of the manipulations embedded in audio, video, or even cross-modal forms. To enhance the modeling of cross-modal forgery clues, we employ audio-visual speech recognition (AVSR) as a preliminary task. This efficiently extracts speech correlations across modalities, a feature challenging for deepfakes to replicate. Additionally, we propose a dual-label detection approach that follows the structure of AVSR to support the independent detection of each modality. Extensive experiments on three audio-visual datasets show that our scheme outperforms state-of-the-art detection methods with promising performance on modality-agnostic audio/video deepfakes.

Audio-Visual Temporal Forgery Detection Using Embedding-Level Fusion and Multi-Dimensional Contrastive Loss

AVoiD-DF: Audio-Visual Joint Learning for Detecting Deepfake

Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization

Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization

AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Video Deepfake Detection

Unified Video and Image Representation for Boosted Video Face Forgery Detection

Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies

Temporal Feature Prediction in Audio–Visual Deepfake Detection

AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection

Audio-Visual Contrastive Pre-train for Face Forgery Detection

AVSecure: an Audio-Visual Watermarking Framework for Proactive Deepfake Detection

AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies

A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection

Refining Localized Attention Features with Multi-Scale Relationships for Enhanced Deepfake Detection in Spatial-Frequency Domain

A Unified Framework for Modality-Agnostic Deepfakes Detection

Transferring Audio Deepfake Detection Capability Across Languages

An Efficient Temporary Deepfake Location Approach Based Embeddings for Partially Spoofed Audio Detection

AVTENet: Audio-Visual Transformer-based Ensemble Network Exploiting Multiple Experts for Video Deepfake Detection

UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization

A defensive attention mechanism to detect deepfake content across multiple modalities

An Audio-Visual Attention Based Multimodal Network for Fake Talking Face Videos Detection