Deep Generation of Face Images from Sketches

Shu-Yu Chen,Wanchao Su,Lin Gao,Shihong Xia,Hongbo Fu
DOI: https://doi.org/10.48550/arXiv.2006.01047
2020-06-05
Abstract:Recent deep image-to-image translation techniques allow fast generation of face images from freehand sketches. However, existing solutions tend to overfit to sketches, thus requiring professional sketches or even edge maps as input. To address this issue, our key idea is to implicitly model the shape space of plausible face images and synthesize a face image in this space to approximate an input sketch. We take a local-to-global approach. We first learn feature embeddings of key face components, and push corresponding parts of input sketches towards underlying component manifolds defined by the feature vectors of face component samples. We also propose another deep neural network to learn the mapping from the embedded component features to realistic images with multi-channel feature maps as intermediate results to improve the information flow. Our method essentially uses input sketches as soft constraints and is thus able to produce high-quality face images even from rough and/or incomplete sketches. Our tool is easy to use even for non-artists, while still supporting fine-grained control of shape details. Both qualitative and quantitative evaluations show the superior generation ability of our system to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.
Graphics,Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The problem that the paper "DeepFaceDrawing: Deep Generation of Face Images from Sketches" attempts to solve is the limitations of the existing deep - learning - based sketch - to - image conversion techniques when dealing with sketch inputs. Specifically, the existing methods often require high - quality professional sketches or edge maps as inputs, which restricts the use by non - professional users. These problems are mainly reflected in the following aspects: 1. **High quality requirements for input sketches**: Most of the existing solutions tend to over - fit the input sketches, which means that they usually require professional sketches or edge maps as inputs. However, it is very difficult for users without professional training to create such sketches. 2. **Lack of flexibility**: The existing methods perform poorly when dealing with rough or incomplete sketches and cannot generate high - quality facial images. 3. **Hard - constraint problems**: The existing methods regard the input sketches as hard constraints, resulting in difficulties in dealing with inconsistencies and errors in the sketches during the synthesis process. To solve these problems, the paper proposes a new deep - learning framework, aiming to generate high - quality facial images from rough or incomplete free - hand - drawn sketches. The key idea of this framework is to implicitly model the possible facial - image - shape space and synthesize facial images through a local - to - global method. Specifically, the paper proposes the following innovations: - **Implicitly modeling the facial - component manifold**: By learning the feature embeddings of facial components, each part of the input sketch is projected onto the corresponding component manifold, thereby improving the rationality of the synthesized image. - **Soft constraints**: The input sketches are regarded as soft constraints, allowing the system to generate higher - quality facial images while maintaining the user's intention. - **Multi - channel feature maps**: By mapping the component feature vectors to multi - channel feature maps, the information flow is improved, so as to better fuse facial components and reduce the inconsistencies in the synthesis results. - **User - friendly**: A shading - guided interface is provided, enabling non - artist users to easily input structurally - reasonable facial sketches as well. In short, the goal of this paper is to develop a method that can generate high - quality facial images from rough or incomplete sketches, while maintaining the user's intention and improving the ease - of - use and expressiveness of the system.