Abstract:Makeup transfer is the task of applying on a source face the makeup style from a reference image. Real-life makeups are diverse and wild, which cover not only color-changing but also patterns, such as stickers, blushes, and jewelries. However, existing works overlooked the latter components and confined makeup transfer to color manipulation, focusing only on light makeup styles. In this work, we propose a holistic makeup transfer framework that can handle all the mentioned makeup components. It consists of an improved color transfer branch and a novel pattern transfer branch to learn all makeup properties, including color, shape, texture, and location. To train and evaluate such a system, we also introduce new makeup datasets for real and synthetic extreme makeup. Experimental results show that our framework achieves the state of the art performance on both light and extreme makeup styles. Code is available at <a class="link-external link-https" href="https://github.com/VinAIResearch/CPM" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the limitations of existing facial makeup transfer techniques when dealing with complex makeup. Specifically, existing methods mainly focus on color conversion and ignore the pattern elements in makeup, such as stickers, blushes, and ornaments. These pattern elements are very common in real - life makeup, but existing methods cannot handle them effectively, resulting in unsatisfactory transfer results. Therefore, this paper proposes a brand - new makeup transfer framework, aiming to handle both color conversion and pattern addition simultaneously to achieve a more comprehensive and realistic makeup transfer effect. This framework can not only handle simple makeup styles but also cope with extremely complex makeup styles, thus significantly expanding the scope of application and the effect of makeup transfer techniques. ### Main contributions of the paper 1. **Comprehensive makeup transfer method**: Consider makeup as a combination of color conversion and pattern addition, and develop a comprehensive makeup transfer method applicable to mild and extreme makeup styles. 2. **Novel network architecture**: Design a new architecture with two branches - a color transfer branch and a pattern transfer branch, and propose to use deformed UV - space human faces when training the two network branches to eliminate differences in shape, head pose, and expression between input human faces. 3. **New makeup transfer data set**: Introduce a new data set containing extreme makeup styles that have not appeared in previous makeup data sets. 4. **Excellent performance**: Through extensive experimental verification, this method has achieved state - of - the - art quantitative and qualitative performance in both mild and extreme makeup transfer tasks. ### Method overview - **Color transfer branch**: Adopt a network structure similar to CycleGAN and use histogram matching to guide color transfer to ensure that the color distribution is consistent with the reference image. - **Pattern transfer branch**: Extract the mask of makeup patterns through supervised learning and perform pattern transfer in the UV space, keeping the shape, texture, and position of the patterns unchanged. - **UV - space processing**: Use PRNet to convert the input image into a UV texture map, eliminating the alignment problem between different human faces, making color and pattern transfer more accurate. ### Data set - **CPM - Real**: A data set containing real human faces and complex makeup, used for testing. - **CPM - Synt - 1**: A data set containing real human faces and synthetic makeup patterns, used for training the pattern transfer branch. - **CPM - Synt - 2**: A data set containing source images, reference images, and ground truth, specifically used for evaluating the effect of pattern transfer. ### Experimental results - **Qualitative experiment**: Experimental results on multiple data sets show that this method outperforms existing methods in both color transfer and pattern transfer, especially when dealing with complex makeup. - **User survey**: Through user surveys, this method also received the highest score in subjective evaluations. - **Quantitative experiment**: Quantitative experiments on labeled data sets further verify the superiority of this method, especially in pattern segmentation and makeup transfer tasks. In conclusion, this paper solves the shortcomings of existing methods in dealing with complex makeup by proposing a comprehensive makeup transfer framework, significantly improving the performance and scope of application of makeup transfer techniques.

Lipstick ain't enough: Beyond Color Matching for In-the-Wild Makeup Transfer

Personalized facial makeup transfer based on outline correspondence

Makeup like a superstar: Deep Localized Makeup Transfer Network

BeautyREC: Robust, Efficient, and Content-preserving Makeup Transfer

BeautyREC: Robust, Efficient, and Component-Specific Makeup Transfer

Stable-Makeup: When Real-World Makeup Transfer Meets Diffusion Model

Improved method of image color transfer.

Example-Based Cosmetic Transfer

AdaCM: Adaptive ColorMLP for Real-Time Universal Photo-realistic Style Transfer

Deep Exemplar-Based Color Transfer for 3D Model

MakeupDiffuse: a double image-controlled diffusion model for exquisite makeup transfer

Spatially-invariant Style-codes Controlled Makeup Transfer

Makeup Transfer Based on Generative Adversarial Network for Large Angle Spatial Misalignment.

Beauty Emakeup: A Deep Makeup Transfer System.

MakeupBag: Disentangling Makeup Extraction and Application

A Multi-Granularity Facial Extreme Makeup Transfer and Removal Model with Local-Global Collaboration

Makeup Transfer Based on Multi-Scale Feature Loss

Facial Makeup Transfer Combining Illumination Transfer

Reference-conditional Makeup-aware Discrimination for Face Image Beautification

A Fine Rendering High-Resolution Makeup Transfer network via inversion-editing strategy

Makeup Based on Segmentation and Local Transfer.