Abstract:Most previous deepfake detection methods bent their efforts to discriminate artifacts by end-to-end training. However, the learned networks often fail to mine the general face forgery information efficiently due to ignoring the data hardness. In this work, we propose to introduce the sample hardness into the training of deepfake detectors via the curriculum learning paradigm. Specifically, we present a novel simple yet effective strategy, named Dynamic Facial Forensic Curriculum (DFFC), which makes the model gradually focus on hard samples during the training. Firstly, we propose Dynamic Forensic Hardness (DFH) which integrates the facial quality score and instantaneous instance loss to dynamically measure sample hardness during the training. Furthermore, we present a pacing function to control the data subsets from easy to hard throughout the training process based on DFH. Comprehensive experiments show that DFFC can improve both within- and cross-dataset performance of various kinds of end-to-end deepfake detectors through a plug-and-play approach. It indicates that DFFC can help deepfake detectors learn general forgery discriminative features by effectively exploiting the information from hard samples.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the deficiency of existing deepfake detection methods in mining general facial forgery information. Specifically, most previous deepfake detection methods distinguish forgery traces through end - to - end training, but these methods fail to efficiently mine general facial forgery information because they ignore the data hardness. This leads to poor generalization performance in real - life scenarios.
To solve this problem, the author proposes a new strategy, called Dynamic Facial Forensic Curriculum (DFFC), which introduces the sample difficulty into the training of deepfake detectors through the curriculum - learning paradigm. DFFC enables the model to gradually focus on more difficult samples during the training process, thus helping the detector to learn forgery features more effectively.
### Main Contributions
1. **Introduction of the curriculum - learning paradigm**: DFFC is the first work to apply the curriculum - learning paradigm to the deepfake detection task and can be used as a plug - and - play module in any end - to - end deepfake detector.
2. **Dynamic evaluation of sample difficulty**: The Dynamic Forensic Hardness (DFH) is proposed, which combines the instantaneous instance loss and the facial quality score to dynamically evaluate the sample difficulty.
3. **Control of the training pace**: A pacing function is designed to gradually control the transition of training data from simple to difficult according to the DFH score, so as to fully mine the forgery information.
### Method Overview
- **Dynamic Forensic Hardness (DFH)**: The sample difficulty is dynamically evaluated by combining the instantaneous instance loss and the facial quality score. The formula is as follows:
\[
DFH_t(i) = d_t(i)+\alpha_f q(i)
\]
where \(d_t(i)\) is the dynamic hardness, \(q(i)\) is the facial quality score, and \(\alpha_f\) is the balance weight.
- **Pacing Function**: Training samples are gradually selected according to the DFH score, starting from simple samples and gradually increasing the difficulty. The algorithm is as follows:
```plaintext
Algorithm 1 Pacing Function
Input: X, t, DFH_t(i), α_f, T = [T0,..., T_N], DA(·), α_k
Output: Sample pool X'_t
...
```
### Experimental Results
Experiments show that DFFC can significantly improve the detection performance of various deepfake detectors on different datasets, including cross - dataset and cross - operation evaluations. In addition, ablation studies show that it is beneficial to introduce both dynamic hardness and facial quality priors in training deepfake detectors.
Through these improvements, DFFC can help deepfake detectors to learn and identify forgery features more effectively, thereby enhancing their generalization ability in real - life scenarios.