Abstract:Given an abstract, deformed, ordinary sketch from untrained amateurs like you and me, this paper turns it into a photorealistic image - just like those shown in Fig. 1(a), all non-cherry-picked. We differ significantly from prior art in that we do not dictate an edgemap-like sketch to start with, but aim to work with abstract free-hand human sketches. In doing so, we essentially democratise the sketch-to-photo pipeline, "picturing" a sketch regardless of how good you sketch. Our contribution at the outset is a decoupled encoder-decoder training paradigm, where the decoder is a StyleGAN trained on photos only. This importantly ensures that generated results are always photorealistic. The rest is then all centred around how best to deal with the abstraction gap between sketch and photo. For that, we propose an autoregressive sketch mapper trained on sketch-photo pairs that maps a sketch to the StyleGAN latent space. We further introduce specific designs to tackle the abstract nature of human sketches, including a fine-grained discriminative loss on the back of a trained sketch-photo retrieval model, and a partial-aware sketch augmentation strategy. Finally, we showcase a few downstream tasks our generation model enables, amongst them is showing how fine-grained sketch-based image retrieval, a well-studied problem in the sketch community, can be reduced to an image (generated) to image retrieval task, surpassing state-of-the-arts. We put forward generated results in the supplementary for everyone to scrutinise.

Automatic genaration of sketch-like pencil drawing from image

Automatic Image-Based Pencil Sketch Rendering

Automatic Generation of Pen-and-ink Drawings from Photos.

PencilArt: A Chromatic Penciling Style Generation Framework.

Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale

Sketch-based 3D Modeling by Aligning Outlines of an Image

A New Approch For Automatic Rendering Of Pen-And-Ink Illustrations

Pencil Drawing Generation Algorithm Based on GMED

Generating Brush Texture for Cursive Style Calligraphy with Autoregressive and Stratified Sampling

DeepFacePencil: Creating Face Images from Freehand Sketches.

Colorsketch: A Drawing Assistant for Generating Color Sketches from Photos

Color Sketch Generation

Example Based Painting Generation

AI supported computer-generated pen-and-ink illustration

Feature-preserving color pencil drawings from photographs

Sketch-to-Art: Synthesizing Stylized Art Images From Sketches

Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

Face photo-sketch portraits transformation via generation pipeline

An Example-Based Facial Sketch Generation System

On-Line Sketch Recognition Using Direction Feature

BézierSketch: A generative model for scalable vector sketches