BeautyBank: Encoding Facial Makeup in Latent Space

Qianwen Lu,Xingchao Yang,Takafumi Taketomi

2024-11-24

Abstract:The advancement of makeup transfer, editing, and image encoding has demonstrated their effectiveness and superior quality. However, existing makeup works primarily focus on low-dimensional features such as color distributions and patterns, limiting their versatillity across a wide range of makeup applications. Futhermore, existing high-dimensional latent encoding methods mainly target global features such as structure and style, and are less effective for tasks that require detailed attention to local color and pattern features of makeup. To overcome these limitations, we propose BeautyBank, a novel makeup encoder that disentangles pattern features of bare and makeup faces. Our method encodes makeup features into a high-dimensional space, preserving essential details necessary for makeup reconstruction and broadening the scope of potential makeup research applications. We also propose a Progressive Makeup Tuning (PMT) strategy, specifically designed to enhance the preservation of detailed makeup features while preventing the inclusion of irrelevant attributes. We further explore novel makeup applications, including facial image generation with makeup injection and makeup similarity measure. Extensive empirical experiments validate that our method offers superior task adaptability and holds significant potential for widespread application in various makeup-related fields. Furthermore, to address the lack of large-scale, high-quality paired makeup datasets in the field, we constructed the Bare-Makeup Synthesis Dataset (BMS), comprising 324,000 pairs of 512x512 pixel images of bare and makeup-enhanced faces.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve several key problems in existing facial makeup processing methods:

1. **Limitations of low - dimensional features**: Existing makeup transfer and editing methods mainly focus on low - dimensional features, such as color distribution and pattern styles. These methods have limited effectiveness in handling complex and diverse makeup applications and are unable to capture sufficient details.

2. **Insufficiency of high - dimensional latent coding**: Existing high - dimensional latent coding methods mainly target global features (such as structure and style) and perform poorly on tasks that require fine local color and pattern features.

3. **Problem of identity feature consistency**: Existing methods often lead to inconsistent facial identity features when generating images with makeup, that is, the generated images have a large deviation from the identity features of the original images.

4. **Interference of irrelevant features**: Existing methods are prone to introduce features irrelevant to makeup (such as hair, background, etc.) when encoding makeup, resulting in a decline in the quality of the generated images.

To solve these problems, the author proposes a new makeup encoder named **BeautyBank** and introduces the following innovations:

- **High - dimensional makeup feature encoding**: BeautyBank encodes makeup features into a high - dimensional latent space, retaining the detailed information required for makeup reconstruction, thereby expanding the application scope of makeup research.

- **Progressive Makeup Tuning strategy (PMT)**: By optimizing makeup encoding in stages, it ensures the retention of detailed makeup features while reducing the influence of irrelevant features.

- **Construction of a large - scale paired dataset**: To address the lack of high - quality large - scale paired makeup datasets, the author constructs the **Bare - Makeup Synthesis Dataset (BMS)**, which contains 324,000 pairs of 512x512 - pixel bare - face and made - up - face images.

### Specific objectives

1. **Improve the accuracy of makeup feature encoding**: Capture more makeup details, such as color, texture, and pattern, through high - dimensional latent space encoding.

2. **Enhance the consistency of identity features**: Ensure that the generated made - up images are consistent with the identity features of the original images and avoid distortion of facial features.

3. **Remove the interference of irrelevant features**: Separate features irrelevant to makeup (such as hair, background, etc.) through a progressive optimization strategy to improve the quality of the generated images.

4. **Support diverse makeup applications**: Including facial image generation, makeup injection, makeup similarity measurement, etc.

Through these improvements, BeautyBank performs well in various makeup - related tasks and has broad application potential.