Abstract:With the rapid development of face recognition (FR) systems, the privacy of face images on social media is facing severe challenges due to the abuse of unauthorized FR systems. Some studies utilize adversarial attack techniques to defend against malicious FR systems by generating adversarial examples. However, the generated adversarial examples, i.e., the protected face images, tend to suffer from subpar visual quality and low transferability. In this paper, we propose a novel face protection approach, dubbed DiffAM, which leverages the powerful generative ability of diffusion models to generate high-quality protected face images with adversarial makeup transferred from reference images. To be specific, we first introduce a makeup removal module to generate non-makeup images utilizing a fine-tuned diffusion model with guidance of textual prompts in CLIP space. As the inverse process of makeup transfer, makeup removal can make it easier to establish the deterministic relationship between makeup domain and non-makeup domain regardless of elaborate text prompts. Then, with this relationship, a CLIP-based makeup loss along with an ensemble attack strategy is introduced to jointly guide the direction of adversarial makeup domain, achieving the generation of protected face images with natural-looking makeup and high black-box transferability. Extensive experiments demonstrate that DiffAM achieves higher visual quality and attack success rates with a gain of 12.98% under black-box setting compared with the state of the arts. The code will be available at
What problem does this paper attempt to address?
### What problems does this paper attempt to solve?
This paper aims to solve the following problems:
1. **Facial privacy protection**: With the rapid development of face recognition (FR) systems, facial images on social media are facing serious privacy challenges. Unauthorized abuse of FR systems may lead to personal privacy leakage. Therefore, there is an urgent need for an effective method to protect facial privacy from unauthorized FR systems.
2. **Limitations of existing methods**:
- **Low visual quality**: The adversarial samples (i.e., protected facial images) generated by existing adversarial attack methods usually have low visual quality.
- **Low transferability**: The attack success rate of the adversarial samples generated by these methods in black - box environments (e.g., commercial APIs) is low.
- **Weaknesses in fine - grained makeup generation**: Text - guided makeup generation methods are difficult to precisely control the details of generated makeup, such as position, color, range, and brightness.
3. **Improving the quality and effectiveness of adversarial makeup generation**: In order to achieve satisfactory makeup generation and good attack effects simultaneously, especially in black - box scenarios, a new method is needed to improve the quality and transferability of adversarial makeup.
### Solutions proposed in the paper
The paper proposes a new method named DiffAM, which utilizes the powerful generation ability of the diffusion model to generate high - quality protected facial images by transferring adversarial makeup from reference images. Specifically, DiffAM contains two modules:
- **Text - guided makeup removal module**: By fine - tuning the diffusion model and combining text prompts in the CLIP space, the makeup in the reference image is removed to generate a non - makeup image. This step simplifies the relationship between the makeup domain and the non - makeup domain and provides precise cross - domain alignment guidance.
- **Image - guided adversarial makeup transfer module**: By combining the makeup loss in the CLIP space and an integrated attack strategy, the generation direction and distance are controlled to generate natural and highly transferable adversarial makeup images.
Through this method, DiffAM can significantly improve the success rate of adversarial attacks while maintaining high visual quality, especially in black - box environments.
### Formula representation
The following are some formulas involved in the paper, presented in Markdown format:
1. **Objective optimization problem**:
\[
\min_{x'} L_{\text{adv}} = D(m(x'), m(x^*))
\]
where \( x' \) is the protected facial image, \( x^* \) is the target facial image, \( m \) represents the feature extractor of the FR model, and \( D(\cdot) \) represents the distance metric.
2. **Text - guided makeup removal loss**:
\[
L_{\text{MR}} = 1-\frac{\Delta I_y\cdot\Delta T}{\|\Delta I_y\|\|\Delta T\|}
\]
where \( \Delta I_y = E_I(\hat{y}(\hat{\theta}_1)) - E_I(y) \), \( \Delta T = E_T(t_{\text{clean}}) - E_T(t_{\text{makeup}}) \), and \( E_I \) and \( E_T \) are the image encoder and text encoder of the CLIP model respectively.
3. **Total loss function**:
\[
L_{\text{total}}=\lambda_{\text{MR}}L_{\text{MR}}+\lambda_{\text{id}}L_{\text{id}}+\lambda_{\text{LPIPS}}L_{\text{LPIPS}}
\]
4. **Makeup direction loss**:
\[
L_{\text{dir}}^{\text{MT}} = 1-\frac{\Delta I_x\cdot\Delta I_{\text{ref}}}{\|\Delta I_