A Quality-Centric Framework for Generic Deepfake Detection

Wentang Song,Zhiyuan Yan,Yuzhen Lin,Taiping Yao,Changsheng Chen,Shen Chen,Yandan Zhao,Shouhong Ding,Bin Li

2024-11-08

Abstract:This paper addresses the generalization issue in deepfake detection by harnessing forgery quality in training data. Generally, the forgery quality of different deepfakes varies: some have easily recognizable forgery clues, while others are highly realistic. Existing works often train detectors on a mix of deepfakes with varying forgery qualities, potentially leading detectors to short-cut the easy-to-spot artifacts from low-quality forgery samples, thereby hurting generalization performance. To tackle this issue, we propose a novel quality-centric framework for generic deepfake detection, which is composed of a Quality Evaluator, a low-quality data enhancement module, and a learning pacing strategy that explicitly incorporates forgery quality into the training process. The framework is inspired by curriculum learning, which is designed to gradually enable the detector to learn more challenging deepfake samples, starting with easier samples and progressing to more realistic ones. We employ both static and dynamic assessments to assess the forgery quality, combining their scores to produce a final rating for each training sample. The rating score guides the selection of deepfake samples for training, with higher-rated samples having a higher probability of being chosen. Furthermore, we propose a novel frequency data augmentation method specifically designed for low-quality forgery samples, which helps to reduce obvious forgery traces and improve their overall realism. Extensive experiments show that our method can be applied in a plug-and-play manner and significantly enhance the generalization performance.

Computer Vision and Pattern Recognition,Cryptography and Security,Machine Learning

What problem does this paper attempt to address?

### Problems the paper attempts to solve

The paper aims to solve the generalization problem in deepfake detection. Specifically, existing deepfake detectors usually contain samples of different forgery qualities in the training data, which may cause the detectors to rely too much on the obvious artifacts in low - quality forgery samples, thus affecting their generalization performance. When encountering cross - domain evaluation (i.e., testing with previously unseen forgery methods or data sources), the performance of these models will drop significantly.

To address this problem, the authors propose a quality - centric framework. By explicitly considering the forgery quality during the training process, the generalization ability of the detector is improved. This framework consists of the following parts:

1. **Quality Evaluator**: It is used to evaluate the forgery quality of each sample.
2. **Low - Quality Data Enhancement Module**: Through the Frequency Data Augmentation (FreDA) method, the obvious artifacts in low - quality samples are reduced and their realism is improved.
3. **Learning Pacing Strategy**: According to the forgery quality of the samples, the training process is gradually adjusted, so that the detector gradually transitions from easy samples to more difficult ones.

### Main contributions

1. **Designed a quality - centric training framework based on curriculum learning**: This framework includes a quality evaluator, a low - quality data enhancement module and a learning pacing strategy, which encourages the model to gradually learn deepfake artifacts according to quality evaluation.
2. **Proposed an evaluation method combining static and dynamic perspectives**: By calculating the Forgery Quality Score (FQS), the difficulty of training samples can be ranked, thus enabling the selection of training samples.
3. **Proposed the Frequency Data Augmentation (FreDA) method**: By reducing the obvious artifacts in low - quality samples and improving their realism instead of directly discarding these samples. FreDA can reduce the artifact traces in forgery samples, and using samples processed by FreDA can enhance the generalization ability of the model.
4. **Extensive experimental verification**: The experimental results show that the proposed framework can significantly improve the generalization performance of various deepfake detectors and can be applied in a plug - and - play manner.

### Related work

1. **Deepfake detection**: Early work mainly focused on hand - crafted features, such as blinking, inconsistent head postures, etc. With the development of deep learning, data - driven detectors show better performance, but still have the problem of poor generalization ability.
2. **Data - level augmentation**: Forgery augmentation techniques improve the generalization ability of detectors by generating diverse forgery samples. However, many methods are still fragile when facing post - processing operations such as compression.
3. **Quality - aware deepfake detection**: Some work attempts to train detectors by combining data of different image qualities to learn quality - independent forgery artifacts. But most existing work focuses on image quality, while this paper focuses on forgery quality.

### Methods

1. **Forgery Quality Score (FQS)**:
- **Static quality**: Calculate the feature similarity between the forgery sample and its corresponding real sample through a pre - trained face recognition network (such as ArcFace). The higher the similarity, the better the sample quality.
- **Dynamic quality**: Evaluate the current difficulty of the sample through the feedback (such as loss value) of the model during the training process. The dynamic quality score is calculated by the moving average method.
- **Composite score**: Weighted sum the static quality and the dynamic quality to get the final Forgery Quality Score (FQS).

2. **Frequency Data Augmentation (FreDA)**:
- Low - quality samples usually have obvious distortion and artifacts. FreDA generates new enhanced samples by retaining the low - frequency information (overall structure) of the real face and combining the high - frequency information (texture details) of the forgery sample, thereby reducing artifacts and improving realism.

3. **Learning Pacing Strategy**:
- According to the value of FQS, gradually adjust the difficulty of the samples used in the training process. Initial stage...