Abstract:Imagining a colored realistic image from an arbitrarily drawn sketch is one of the human capabilities that we eager machines to mimic. Unlike previous methods that either requires the sketch-image pairs or utilize low-quantity detected edges as sketches, we study the exemplar-based sketch-to-image (s2i) synthesis task in a self-supervised learning manner, eliminating the necessity of the paired sketch data. To this end, we first propose an unsupervised method to efficiently synthesize line-sketches for general RGB-only datasets. With the synthetic paired-data, we then present a self-supervised Auto-Encoder (AE) to decouple the content/style features from sketches and RGB-images, and synthesize images that are both content-faithful to the sketches and style-consistent to the RGB-images. While prior works employ either the cycle-consistence loss or dedicated attentional modules to enforce the content/style fidelity, we show AE's superior performance with pure self-supervisions. To further improve the synthesis quality in high resolution, we also leverage an adversarial network to refine the details of synthetic images. Extensive experiments on 1024*1024 resolution demonstrate a new state-of-art-art performance of the proposed model on CelebA-HQ and Wiki-Art datasets. Moreover, with the proposed sketch generator, the model shows a promising performance on style mixing and style transfer, which require synthesized images to be both style-consistent and semantically meaningful. Our code is available on <a class="link-external link-https" href="https://github.com/odegeasslbc/Self-Supervised-Sketch-to-Image-Synthesis-PyTorch" rel="external noopener nofollow">this https URL</a>, and please visit <a class="link-external link-https" href="https://create.playform.io/my-projects?mode=sketch" rel="external noopener nofollow">this https URL</a> for an online demo of our model.

Unsupervised Sketch-to-Photo Synthesis

Unsupervised Scene Sketch to Photo Synthesis

Unsupervised Sketch-to-Photo Synthesis Supplementary Material

Text-Guided Scene Sketch-to-Photo Synthesis

Self-Supervised Sketch-to-Image Synthesis

Staged Sketch-to-Image Synthesis via Semi-supervised Generative Adversarial Networks

Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Training

Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis

Unsupervised Facial Geometry Learning for Sketch to Photo Synthesis

Learning to Sketch with Shortcut Cycle Consistency

Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

Unpaired Image-to-Sketch Translation Network for Sketch Synthesis

Cross-Domain Face Sketch Synthesis.

Quality Guided Sketch-to-Photo Image Synthesis

Adversarial Open Domain Adaption Framework (AODA): Sketch-to-Photo Synthesis

Sketch-to-Art: Synthesizing Stylized Art Images From Sketches

End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning

Face Sketch–Photo Synthesis and Retrieval Using Sparse Representation

Face sketch-to-photo synthesis from simple line drawing.

Semi-supervised Learning for Face Sketch Synthesis in the Wild

Learning semantic priors for texture-realistic sketch-to-image synthesis