Portrait Stylization: Artistic Style Transfer with Auxiliary Networks for Human Face Stylization

Thiago Ambiel

2023-09-24

Abstract:Today's image style transfer methods have difficulty retaining humans face individual features after the whole stylizing process. This occurs because the features like face geometry and people's expressions are not captured by the general-purpose image classifiers like the VGG-19 pre-trained models. This paper proposes the use of embeddings from an auxiliary pre-trained face recognition model to encourage the algorithm to propagate human face features from the content image to the final stylized result.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that during the image style transfer process, facial features (such as facial geometric structure and expressions) cannot be effectively preserved. Specifically, existing style transfer methods, when dealing with images containing human faces, are prone to cause face distortion, making the faces in the finally generated images difficult to recognize. This is mainly because commonly used pre - trained models (such as VGG - 19) cannot capture important features of the human face.

### Background of the Paper
Style transfer technology is a method that combines a given content image (such as an animal, landscape or portrait) with a given style image (such as an oil painting or abstract art) to generate a final stylized result image. Although this technology has made remarkable progress, it still has limitations when dealing with images containing human faces. These methods usually lead to face distortion, especially when generating classic portraits such as "Mona Lisa" or "Self - Portrait".

### Main Problem
The main problem with existing methods is that the general - purpose image classifiers (such as VGG - 19) they use cannot capture important features of the human face, such as facial geometric structure and expressions. Therefore, these features are easily lost during the stylization process, resulting in severe face distortion in the generated images.

### Solution
To solve this problem, this paper proposes a new method, called "Portrait Stylization". This method extracts facial features by introducing an auxiliary pre - trained face recognition model (such as FaceNet) and uses them as part of the loss function to encourage the algorithm to preserve facial features during the stylization process.

### Specific Improvements
1. **FaceID Loss**: A new loss function term, called FaceID Loss, is introduced to calculate the facial feature differences between the content image and the result image. This loss function is based on the facial features extracted by the pre - trained FaceNet model.
2. **FaceMesh Loss**: Further introduce the FaceMesh algorithm to calculate the differences in the facial surface geometric structure between the content image and the result image. This helps to better preserve the geometric structure of the face.
3. **Support for Multiple Faces**: In order to support images containing multiple faces, a face detection algorithm (such as MTCNN) is used to extract face coordinates and perform stylization processing on each face separately.

### Experimental Results
The experimental results show that, compared with existing methods, this method can significantly reduce face distortion when dealing with images containing human faces while maintaining the stylized expression effect. In addition, this method can also be applied to other types of images. When the face feature loss is not used, its performance is comparable to that of existing methods.

### Conclusion
This paper successfully solves the problem of face feature loss during the style transfer process by introducing an auxiliary face recognition model. This method not only improves the quality of the generated images but also provides a new direction for future research, such as exploring the application of other domain - specific models in stylization.