Personalized hairstyle and hair color editing based on multi-feature fusion

Jiayi Xu,Chenming Zhang,Weikang Zhu,Hongbin Zhang,Li Li,Xiaoyang Mao
DOI: https://doi.org/10.1007/s00371-024-03468-2
IF: 2.835
2024-05-31
The Visual Computer
Abstract:In the metaverse era, virtual design of hairstyle becomes very popular for personalized aesthetics. As hair design tasks can be decomposed into hair attribute editing and generation, the development of generative adversarial networks (GANs) has significantly prompted its development. The majority of the existing algorithms focus on transferring the overall hair region from one face to another, which ignore fine control over the color and geometric features. Furthermore, these algorithms may result in unnatural generation results. In this paper, we propose a hair modification framework that learns hairstyle information from a reference face mask and color information from a guidance face image. Firstly, the features of the input face image and reference images are extracted through a group of encoders, and then divided into feature vectors of coarse, medium, and fine levels. Secondly, multi-level feature vectors are fused in the latent space using attention-based modulation modules. Finally, the fused feature vector is passed through a StyleGAN generator to generate face images with specified hairstyle and hair color. Experimental results show that the proposed method can finely simulate the hairstyle transition between long and short hair under the constraint of the reference mask, and can produce realistic fusion effects in the hair-covered regions, such as ears, neck, and forehead. Various hair dyeing effects that adapt to personalized characteristics are demonstrated, as facial features including skin color and hair texture are preserved when transferring the hair color.
computer science, software engineering
What problem does this paper attempt to address?
This paper attempts to address the problem of achieving personalized hairstyle and hair color editing in virtual design, specifically how to achieve fine control over hairstyle and hair color while preserving irrelevant facial features. Existing algorithms mainly focus on transferring the overall hairstyle from one face to another but neglect fine-grained control over color and geometric features, resulting in unnatural generation outcomes. Therefore, this paper proposes a hair modification framework based on multi-feature fusion, aiming to learn hairstyle and hair color information through reference masks and guiding images, thereby achieving more natural and refined hairstyle and hair color editing. Specifically, this method is implemented through the following steps: 1. **Feature Extraction**: Three encoders are used to extract features from the input face image, hairstyle reference mask, and hair color reference image, and these features are divided into coarse, medium, and fine levels. 2. **Feature Fusion**: A modulation module based on the attention mechanism is used to fuse multi-level feature vectors in the latent space. 3. **Generate New Image**: The fused feature vectors are input into a pre-trained StyleGAN generator to generate face images with the specified hairstyle and hair color. Through this method, the paper addresses the shortcomings of existing algorithms in hairstyle and hair color editing, achieving more natural hair fusion effects and personalized hairstyle and hair color modifications.