Abstract:Sketch-based face recognition is an interesting task in vision and multimedia research, yet it is quite challenging due to the great difference between face photos and sketches. In this paper, we propose a novel approach for photo-sketch generation, aiming to automatically transform face photos into detail-preserving personal sketches. Unlike the traditional models synthesizing sketches based on a dictionary of exemplars, we develop a fully convolutional network to learn the end-to-end photo-sketch mapping. Our approach takes whole face photos as inputs and directly generates the corresponding sketch images with efficient inference and learning, in which the architecture are stacked by only convolutional kernels of very small sizes. To well capture the person identity during the photo-sketch transformation, we define our optimization objective in the form of joint generative-discriminative minimization. In particular, a discriminative regularization term is incorporated into the photo-sketch generation, enhancing the discriminability of the generated person sketches against other individuals. Extensive experiments on several standard benchmarks suggest that our approach outperforms other state-of-the-art methods in both photo-sketch generation and face sketch verification.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve automatic conversion from photos to sketches in the case where there are significant differences between face photos and sketches. Specifically, the author proposes a new method to learn the end - to - end mapping relationship from photos to sketches through the Fully Convolutional Network (FCN), aiming to automatically generate face sketches with detail preservation. This method is different from the traditional method of synthesizing sketches based on dictionary examples. It can directly take the entire face photo as input, efficiently generate the corresponding sketch image, and maintain high efficiency during the learning and inference processes. ### Background of the Paper and Problem Description - **Problem Background**: Sketch - style face recognition is an interesting but extremely challenging task, mainly because of the huge differences between face photos and sketches. These differences lead to the fact that traditional photo - based face verification methods cannot be directly applied to sketch - based face verification. - **Limitations of Existing Methods**: Most of the existing automatic face - sketch generation methods use dictionary - based synthesis methods, that is, synthesizing new sketches through a series of predefined examples. These methods are inefficient when dealing with large - scale data and it is difficult to preserve the details of personal characteristics. - **Research Objectives**: This paper proposes a new method based on the Fully Convolutional Network to directly generate detail - preserved personal sketches from face photos while maintaining high recognition of personal identity during the generation process. ### Innovation Points of the Method 1. **Application of Fully Convolutional Network (FCN)**: A network architecture composed entirely of convolutional layers is proposed to learn the end - to - end mapping from photos to sketches. This architecture can handle complex non - linear problems and produce pixel - level outputs, which is very suitable for the photo - to - sketch generation task. 2. **Joint Generation - Discrimination Optimization Objective**: An optimization objective function including the generation loss and the discrimination regularization term is defined. The generation loss ensures that the generated sketch is as close as possible to the real sketch, while the discrimination regularization term enhances the discrimination ability of the generated sketches among different individuals. 3. **Experimental Verification**: Through extensive experiments on multiple standard benchmark datasets, it is proved that this method is superior to existing methods in both photo - to - sketch generation and face - sketch verification tasks. ### Main Contributions 1. **For the first time, an end - to - end photo - to - sketch generation model based on the Fully Convolutional Network is proposed**. 2. **The joint generation - discrimination optimization objective is introduced**, which improves the detail - preservation ability and the discrimination ability among individuals of the generated sketches. 3. **Superior performance is achieved on multiple benchmark datasets**, demonstrating the effectiveness and robustness of this method. ### Conclusion This paper successfully solves the limitations of traditional methods in generating detail - preserved personal sketches by proposing an end - to - end photo - to - sketch generation method based on the Fully Convolutional Network. The experimental results show that this method is not only superior to existing methods in terms of generation quality, but also has higher efficiency when dealing with large - scale data.

End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning

A Sketch-Transformer Network for Face Photo-Sketch Synthesis

Recognizing Facial Sketches by Generating Photorealistic Faces Guided by Descriptive Attributes

Content-Adaptive Sketch Portrait Generation by Decompositional Representation Learning

Deep Generation of Face Images from Sketches

Face Sketch-Photo Synthesis under Multi-dictionary Sparse Representation Framework

A Deep Collaborative Framework for Face Photo–Sketch Synthesis

Face Photo–sketch Synthesis Via Intra-Domain Enhancement

Sketch Face Recognition Via Cascaded Transformation Generation Network

Cross-Domain Face Sketch Synthesis.

Sketch-to-photo face generation based on semantic consistency preserving and similar connected component refinement

Deep Graphical Feature Learning for Face Sketch Synthesis.

Face Sketch-to-photo Transformation with Multi-Scale Self-Attention GAN.

Recognizing Minimal Facial Sketch by Generating Photorealistic Faces with the Guidance of Descriptive Attributes

Biphasic Face Photo-Sketch Synthesis via Semantic-Driven Generative Adversarial Network with Graph Representation Learning

Face photo-line drawings synthesis based on local extraction preserving generative adversarial networks

High-Fidelity Face Sketch-To-Photo Synthesis Using Generative Adversarial Network

DeepFaceDrawing: Deep Generation of Face Images from Sketches

Towards Realistic Face Photo-Sketch Synthesis via Composition-Aided GANs

Face Sketch Synthesis in the Wild Via Deep Patch Representation-Based Probabilistic Graphical Model.

Face Sketch Synthesis Via Sparse Representation-Based Greedy Search