Abstract:Imagining a colored realistic image from an arbitrarily drawn sketch is one of the human capabilities that we eager machines to mimic. Unlike previous methods that either requires the sketch-image pairs or utilize low-quantity detected edges as sketches, we study the exemplar-based sketch-to-image (s2i) synthesis task in a self-supervised learning manner, eliminating the necessity of the paired sketch data. To this end, we first propose an unsupervised method to efficiently synthesize line-sketches for general RGB-only datasets. With the synthetic paired-data, we then present a self-supervised Auto-Encoder (AE) to decouple the content/style features from sketches and RGB-images, and synthesize images that are both content-faithful to the sketches and style-consistent to the RGB-images. While prior works employ either the cycle-consistence loss or dedicated attentional modules to enforce the content/style fidelity, we show AE's superior performance with pure self-supervisions. To further improve the synthesis quality in high resolution, we also leverage an adversarial network to refine the details of synthetic images. Extensive experiments on 1024*1024 resolution demonstrate a new state-of-art-art performance of the proposed model on CelebA-HQ and Wiki-Art datasets. Moreover, with the proposed sketch generator, the model shows a promising performance on style mixing and style transfer, which require synthesized images to be both style-consistent and semantically meaningful. Our code is available on <a class="link-external link-https" href="https://github.com/odegeasslbc/Self-Supervised-Sketch-to-Image-Synthesis-PyTorch" rel="external noopener nofollow">this https URL</a>, and please visit <a class="link-external link-https" href="https://create.playform.io/my-projects?mode=sketch" rel="external noopener nofollow">this https URL</a> for an online demo of our model.

Deep Self-Supervised Representation Learning for Free-Hand Sketch

Sketch-R2CNN: an Attentive Network for Vector Sketch Recognition

Deep Learning for Free-Hand Sketch: A Survey

Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

<i>Sketch-R2CNN</i>: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

On Learning Semantic Representations for Large-Scale Abstract Sketches

Stroke-based semantic segmentation for scene-level free-hand sketches

SFSegNet: Parse Freehand Sketches using Deep Fully Convolutional Networks

Self-Supervised Sketch-to-Image Synthesis

Sketch-Specific Data Augmentation for Freehand Sketch Recognition

DeepFacePencil: Creating Face Images from Freehand Sketches.

Semi-supervised Learning for Face Sketch Synthesis in the Wild

FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context

Learning Deep Sketch Abstraction

Deep CNN-based Features for Hand-Drawn Sketch Recognition Via Transfer Learning Approach

End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning

Deep3DSketch: 3D modeling from Free-hand Sketches with View- and Structural-Aware Adversarial Training

Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition

SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation

Deep Generation of Face Images from Sketches