Face Reconstruction from Face Embeddings using Adapter to a Face Foundation Model

Hatef Otroshi Shahreza,Anjith George,Sébastien Marcel

2024-11-06

Abstract:Face recognition systems extract embedding vectors from face images and use these embeddings to verify or identify individuals. Face reconstruction attack (also known as template inversion) refers to reconstructing face images from face embeddings and using the reconstructed face image to enter a face recognition system. In this paper, we propose to use a face foundation model to reconstruct face images from the embeddings of a blackbox face recognition model. The foundation model is trained with 42M images to generate face images from the facial embeddings of a fixed face recognition model. We propose to use an adapter to translate target embeddings into the embedding space of the foundation model. The generated images are evaluated on different face recognition models and different datasets, demonstrating the effectiveness of our method to translate embeddings of different face recognition models. We also evaluate the transferability of reconstructed face images when attacking different face recognition models. Our experimental results show that our reconstructed face images outperform previous reconstruction attacks against face recognition models.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to reconstruct facial images from the embedding vectors of facial recognition systems in a black - box scenario, and use these reconstructed facial images to attack facial recognition systems. Specifically, the author proposes a new method that utilizes the foundation model to reconstruct facial images from the embedding vectors of different facial recognition models. To achieve this goal, the author introduces an adapter module, which can map the target embedding vectors to the input space of the foundation model, so that the foundation model can generate corresponding facial images. This method not only reduces the computational resource requirements for attacking facial recognition systems but also improves the quality of reconstructed facial images and the attack success rate.

### Main Contributions
1. **Adapter Module**: A simple and effective adapter module is proposed to map facial embedding vectors to the input space of the foundation model. This enables the foundation model to process embedding vectors from different facial recognition models without retraining the foundation model for each new facial recognition model.
2. **New Facial Reconstruction Attack Method**: Use the foundation model for facial reconstruction attacks, which is the first facial recognition system attack method based on the foundation model.
3. **Extensive Experimental Verification**: Through experiments on multiple facial recognition datasets, the effectiveness of reconstructed facial images is verified, and the transferability of these images between different facial recognition models is evaluated. The experimental results show that this method outperforms existing facial reconstruction attack methods in attacking facial recognition systems.

### Method Overview
- **Threat Model**:
- **Attacker's Goal**: Reconstruct facial images from leaked facial embedding vectors and use these images to enter the target facial recognition system.
- **Attacker's Knowledge**: The attacker knows the facial embedding vectors leaked from the victim's facial recognition system database and has black - box access (such as SDK, API, etc.).
- **Attacker's Ability**: The attacker can use the reconstructed facial images to enter the target facial recognition system.
- **Attacker's Strategy**: Use the foundation model to reconstruct facial images from the leaked embedding vectors and then use these images to attack the target facial recognition system.

- **Facial Reconstruction**:
- Utilize the Arc2Face foundation model, which is trained on large - scale data and can generate consistent identity images from the embedding vectors of a specific facial recognition model.
- Introduce an adapter module to map the embedding vectors of different facial recognition models to the input space of the foundation model.

### Experimental Results
- **Black - Box Attack**: Black - box attack experiments were carried out on multiple facial recognition models and datasets, and the results show that the successful attack rate of this method on different facial recognition models is higher than that of existing methods.
- **Transferability Evaluation**: The transferability of the reconstructed facial images in different facial recognition systems was evaluated, and the results show that these images can be effectively transferred between different facial recognition models.

### Conclusion
This paper proposes a new facial reconstruction attack method. By using the foundation model and adapter module, it can efficiently reconstruct high - quality facial images from the embedding vectors of facial recognition systems in a black - box scenario and successfully attack facial recognition systems. The experimental results verify the effectiveness and superiority of this method.