Abstract:Deepfakes have recently raised significant trust issues and security concerns among the public. Compared to CNN face forgery detectors, ViT-based methods take advantage of the expressivity of transformers, achieving superior detection performance. However, these approaches still exhibit the following limitations: (1) Fully fine-tuning ViT-based models from ImageNet weights demands substantial computational and storage resources; (2) ViT-based methods struggle to capture local forgery clues, leading to model bias; (3) These methods limit their scope on only one or few face forgery features, resulting in limited generalizability. To tackle these challenges, this work introduces Mixture-of-Experts modules for Face Forgery Detection (MoE-FFD), a generalized yet parameter-efficient ViT-based approach. MoE-FFD only updates lightweight Low-Rank Adaptation (LoRA) and Adapter layers while keeping the ViT backbone frozen, thereby achieving parameter-efficient training. Moreover, MoE-FFD leverages the expressivity of transformers and local priors of CNNs to simultaneously extract global and local forgery clues. Additionally, novel MoE modules are designed to scale the model's capacity and smartly select optimal forgery experts, further enhancing forgery detection performance. Our proposed learning scheme can be seamlessly adapted to various transformer backbones in a plug-and-play manner. Extensive experimental results demonstrate that the proposed method achieves state-of-the-art face forgery detection performance with significantly reduced parameter overhead. The code is released at: <a class="link-external link-https" href="https://github.com/LoveSiameseCat/MoE-FFD" rel="external noopener nofollow">this https URL</a>.

Towards Generalizable and Robust Face Manipulation Detection Via Bag-of-feature

Towards Generalizable and Robust Face Manipulation Detection via Bag-of-local-feature

Robust Face Recognition by Fusion Local Singular Value Feature and Deformable Model

Towards Mask-robust Face Recognition.

Learning Forgery Region-Aware and ID-Independent Features for Face Manipulation Detection

Semantic and Frequency Representation Mining for Face Manipulation Detection

Detecting facial manipulated images via one-class domain generalization

Hierarchical Frequency-Assisted Interactive Networks for Face Manipulation Detection

Exploring Depth Information for Detecting Manipulated Face Videos

Face Manipulation Detection Based on Supervised Multi-Feature Fusion Attention Network

Learning to mask: Towards generalized face forgery detection

Mining Generalized Features for Detecting AI-Manipulated Fake Faces

Robust Face-Swap Detection Based on 3D Facial Shape Information

Exploiting Facial Relationships and Feature Aggregation for Multi-Face Forgery Detection

Extracting deep local features to detect manipulated images of human faces

Multi-feature fusion based face forgery detection with local and global characteristics

Unified Video and Image Representation for Boosted Video Face Forgery Detection

Face Forgery Detection with Long-Range Noise Features and Multilevel Frequency-Aware Clues

Enhancing General Face Forgery Detection via Vision Transformer with Low-Rank Adaptation

Recurrent Convolutional Strategies for Face Manipulation Detection in Videos

MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection