Abstract:Given an abstract, deformed, ordinary sketch from untrained amateurs like you and me, this paper turns it into a photorealistic image - just like those shown in Fig. 1(a), all non-cherry-picked. We differ significantly from prior art in that we do not dictate an edgemap-like sketch to start with, but aim to work with abstract free-hand human sketches. In doing so, we essentially democratise the sketch-to-photo pipeline, "picturing" a sketch regardless of how good you sketch. Our contribution at the outset is a decoupled encoder-decoder training paradigm, where the decoder is a StyleGAN trained on photos only. This importantly ensures that generated results are always photorealistic. The rest is then all centred around how best to deal with the abstraction gap between sketch and photo. For that, we propose an autoregressive sketch mapper trained on sketch-photo pairs that maps a sketch to the StyleGAN latent space. We further introduce specific designs to tackle the abstract nature of human sketches, including a fine-grained discriminative loss on the back of a trained sketch-photo retrieval model, and a partial-aware sketch augmentation strategy. Finally, we showcase a few downstream tasks our generation model enables, amongst them is showing how fine-grained sketch-based image retrieval, a well-studied problem in the sketch community, can be reduced to an image (generated) to image retrieval task, surpassing state-of-the-arts. We put forward generated results in the supplementary for everyone to scrutinise.

PhotoSketch: Internet Image Montage

Sketch2Photo: Internet Image Montage

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

An on-line sketch recognition algorithm for composite shape

Sketch-Based Retrieval in Large-Scale Image Database Via Position-Aware Silhouette Matching.

Fashionsketch: An Interactive Sketch-Based Massive Image Retrieval For Fashion E-Business

Unsupervised Scene Sketch to Photo Synthesis

Sketch realizing: Lifelike portrait synthesis from sketch

Colorsketch: A Drawing Assistant for Generating Color Sketches from Photos

Automatic Image-Based Pencil Sketch Rendering

Image-based facial sketch-to-photo synthesis via online coupled dictionary learning

Image based Face Sketch-to-Photo via Online Coupled Dictionary Learning

Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models

Photo-sketch Synthesis and Recognition Based on Subspace Learning

Recognition Of Sketches In Photos

Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition

Face Sketch–Photo Synthesis and Retrieval Using Sparse Representation

FaceShop: Deep Sketch-based Face Image Editing

CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing

Generative Photomontage