Abstract:With the rapid progress of generative models, the current challenge in face forgery detection is how to effectively detect realistic manipulated faces from different unseen domains. Though previous studies show that pre-trained Vision Transformer (ViT) based models can achieve some promising results after fully fine-tuning on the Deepfake dataset, their generalization performances are still unsatisfactory. One possible reason is that fully fine-tuned ViT-based models may disrupt the pre-trained features [1, 2] and overfit to some data-specific patterns [3]. To alleviate this issue, we present a \textbf{F}orgery-aware \textbf{A}daptive \textbf{Vi}sion \textbf{T}ransformer (FA-ViT) under the adaptive learning paradigm, where the parameters in the pre-trained ViT are kept fixed while the designed adaptive modules are optimized to capture forgery features. Specifically, a global adaptive module is designed to model long-range interactions among input tokens, which takes advantage of self-attention mechanism to mine global forgery clues. To further explore essential local forgery clues, a local adaptive module is proposed to expose local inconsistencies by enhancing the local contextual association. In addition, we introduce a fine-grained adaptive learning module that emphasizes the common compact representation of genuine faces through relationship learning in fine-grained pairs, driving these proposed adaptive modules to be aware of fine-grained forgery-aware information. Extensive experiments demonstrate that our FA-ViT achieves state-of-the-arts results in the cross-dataset evaluation, and enhances the robustness against unseen perturbations. Particularly, FA-ViT achieves 93.83\% and 78.32\% AUC scores on Celeb-DF and DFDC datasets in the cross-dataset evaluation. The code and trained model have been released at: <a class="link-external link-https" href="https://github.com/LoveSiameseCat/FAViT" rel="external noopener nofollow">this https URL</a>.

Unified Video and Image Representation for Boosted Video Face Forgery Detection

UniForensics: Face Forgery Detection via General Facial Representation

Video Forgery Detection Using Spatio-Temporal Dual Transformer.

Temporal Consistency Based Deep Face Forgery Detection Network.

Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer

Towards General Visual-Linguistic Face Forgery Detection.

Detection of Deepfake Videos Using Long-Distance Attention

UVL2: A Unified Framework for Video Tampering Localization

UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization

Object-Based Video Forgery Detection Via Dual-Stream Networks.

Exploring Bi-Level Inconsistency Via Blended Images for Generalizable Face Forgery Detection

A Temporal Consistency Learning Framework for Face Forgery Detection

Spatial-Temporal Frequency Forgery Clue for Video Forgery Detection in VIS and NIR Scenario.

UIA-ViT: Unsupervised Inconsistency-Aware Method Based on Vision Transformer for Face Forgery Detection.

Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-modal Manipulation

Exploring Temporal Coherence for More General Video Face Forgery Detection

Attention-guided Fine-grained Feature Learning For Robust Face Forgery Detection

Collaborative Feature Learning for Fine-grained Facial Forgery Detection and Segmentation

Deep Visual Identity Forgery and Detection

A Dual Descriptor Combined with Frequency Domain Reconstruction Learning for Face Forgery Detection in Deepfake Videos

VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping