Abstract:Imagining a colored realistic image from an arbitrarily drawn sketch is one of the human capabilities that we eager machines to mimic. Unlike previous methods that either requires the sketch-image pairs or utilize low-quantity detected edges as sketches, we study the exemplar-based sketch-to-image (s2i) synthesis task in a self-supervised learning manner, eliminating the necessity of the paired sketch data. To this end, we first propose an unsupervised method to efficiently synthesize line-sketches for general RGB-only datasets. With the synthetic paired-data, we then present a self-supervised Auto-Encoder (AE) to decouple the content/style features from sketches and RGB-images, and synthesize images that are both content-faithful to the sketches and style-consistent to the RGB-images. While prior works employ either the cycle-consistence loss or dedicated attentional modules to enforce the content/style fidelity, we show AE's superior performance with pure self-supervisions. To further improve the synthesis quality in high resolution, we also leverage an adversarial network to refine the details of synthetic images. Extensive experiments on 1024*1024 resolution demonstrate a new state-of-art-art performance of the proposed model on CelebA-HQ and Wiki-Art datasets. Moreover, with the proposed sketch generator, the model shows a promising performance on style mixing and style transfer, which require synthesized images to be both style-consistent and semantically meaningful. Our code is available on <a class="link-external link-https" href="https://github.com/odegeasslbc/Self-Supervised-Sketch-to-Image-Synthesis-PyTorch" rel="external noopener nofollow">this https URL</a>, and please visit <a class="link-external link-https" href="https://create.playform.io/my-projects?mode=sketch" rel="external noopener nofollow">this https URL</a> for an online demo of our model.

Adversarial Open Domain Adaption Framework (AODA): Sketch-to-Photo Synthesis

Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis

High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks

Self-Supervised Sketch-to-Image Synthesis

Unsupervised Scene Sketch to Photo Synthesis

Staged Sketch-to-Image Synthesis via Semi-supervised Generative Adversarial Networks

Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

Composition-Aided Face Photo-Sketch Synthesis.

SmartPaint: a Co-Creative Drawing System Based on Generative Adversarial Networks

Creative Sketch Generation

Quality Guided Sketch-to-Photo Image Synthesis

SketchyCOCO: Image Generation from Freehand Scene Sketches

Deep pelican based synthesis model for photo-sketch face synthesis and recognition

Sketch-Specific Data Augmentation for Freehand Sketch Recognition

Towards Realistic Face Photo-Sketch Synthesis via Composition-Aided GANs

Scribbler: Controlling Deep Image Synthesis with Sketch and Color

Face Sketch Synthesis via Semantic-Driven Generative Adversarial Network

Biphasic Face Photo-Sketch Synthesis via Semantic-Driven Generative Adversarial Network with Graph Representation Learning

Toward Realistic Face Photo–Sketch Synthesis via Composition-Aided GANs

Learning to Sketch with Shortcut Cycle Consistency

Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Training