FRIDAY: Mitigating Unintentional Facial Identity in Deepfake Detectors Guided by Facial Recognizers

Younhun Kim,Myung-Joon Kwon,Wonjun Lee,Changick Kim

2024-12-19

Abstract:Previous Deepfake detection methods perform well within their training domains, but their effectiveness diminishes significantly with new synthesis techniques. Recent studies have revealed that detection models often create decision boundaries based on facial identity rather than synthetic artifacts, resulting in poor performance on cross-domain datasets. To address this limitation, we propose Facial Recognition Identity Attenuation (FRIDAY), a novel training method that mitigates facial identity influence using a face recognizer. Specifically, we first train a face recognizer using the same backbone as the Deepfake detector. The recognizer is then frozen and employed during the detector's training to reduce facial identity information. This is achieved by feeding input images into both the recognizer and the detector, and minimizing the similarity of their feature embeddings through our Facial Identity Attenuating loss. This process encourages the detector to generate embeddings distinct from the recognizer, effectively reducing the impact of facial identity. Extensive experiments demonstrate that our approach significantly enhances detection performance on both in-domain and cross-domain datasets.

Computer Vision and Pattern Recognition,Cryptography and Security

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: Existing Deepfake detection methods perform well in the training data set (i.e., the in - domain data set), but when faced with data sets generated by new synthesis techniques (i.e., cross - domain data sets), their effectiveness drops significantly. Research shows that the detection model relies on facial identity features rather than synthetic artifacts in the decision - making process, which leads to poor cross - domain performance.

Specifically, the paper points out that when the model makes predictions on cross - domain data sets, due to differences in training data, image quality, and manipulation techniques, the effective features are significantly reduced. Therefore, it is crucial to ensure that the model focuses on artifact - related features during the training process. However, recent research has found that the model inadvertently learns facial identity features during the training process because creating decision boundaries based on facial identity is easier to minimize the loss than based on Deepfake artifacts. This inadvertent facial identity learning is not a big problem in the in - domain data set because there are other useful features; but in the cross - domain data set, the limited effective features are further reduced, resulting in performance degradation.

To solve this problem, the authors propose a novel training method named FRIDAY, which aims to attenuate the facial identity information in the Deepfake detector by using a face recognizer. The specific implementation of FRIDAY is as follows:

1. **Train the face recognizer**: First, train a face recognizer using the same backbone network as the Deepfake detector.
2. **Freeze and utilize the face recognizer**: Then freeze this recognizer and use it to attenuate the facial identity information during the training process of the Deepfake detector. This is achieved by sending the input image to both the recognizer and the detector simultaneously and minimizing the similarity of their feature embeddings. The specific formula is:
\[
L_{\text{fia}}=\left|\frac{z_f \cdot z_d}{\|z_f\|_2\|z_d\|_2}\right|
\]
where \(z_f = f(x)\) is the embedding of the face recognizer and \(z_d = d(x)\) is the embedding of the Deepfake detector.
3. **Composite loss function**: Finally, train the Deepfake detector by combining the classification loss and the facial identity attenuation loss:
\[
L_{\text{total}}=L_{\text{cls}}+\lambda \cdot L_{\text{fia}}
\]
where \(L_{\text{cls}}\) is the classification loss and \(\lambda\) is a weighting factor that balances the importance of the two loss terms.

Through this method, FRIDAY effectively reduces the dependence of the Deepfake detector on facial identity features, thereby improving its detection performance on both in - domain and cross - domain data sets. Experimental results show that FRIDAY outperforms the existing state - of - the - art methods on multiple data sets.