Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

Subhadeep Koley,Ayan Kumar Bhunia,Aneeshan Sain,Pinaki Nath Chowdhury,Tao Xiang,Yi-Zhe Song

2023-03-30

Abstract:Given an abstract, deformed, ordinary sketch from untrained amateurs like you and me, this paper turns it into a photorealistic image - just like those shown in Fig. 1(a), all non-cherry-picked. We differ significantly from prior art in that we do not dictate an edgemap-like sketch to start with, but aim to work with abstract free-hand human sketches. In doing so, we essentially democratise the sketch-to-photo pipeline, "picturing" a sketch regardless of how good you sketch. Our contribution at the outset is a decoupled encoder-decoder training paradigm, where the decoder is a StyleGAN trained on photos only. This importantly ensures that generated results are always photorealistic. The rest is then all centred around how best to deal with the abstraction gap between sketch and photo. For that, we propose an autoregressive sketch mapper trained on sketch-photo pairs that maps a sketch to the StyleGAN latent space. We further introduce specific designs to tackle the abstract nature of human sketches, including a fine-grained discriminative loss on the back of a trained sketch-photo retrieval model, and a partial-aware sketch augmentation strategy. Finally, we showcase a few downstream tasks our generation model enables, amongst them is showing how fine-grained sketch-based image retrieval, a well-studied problem in the sketch community, can be reduced to an image (generated) to image retrieval task, surpassing state-of-the-arts. We put forward generated results in the supplementary for everyone to scrutinise.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to generate realistic photos from abstract hand - drawn sketches. Specifically, existing methods can generate high - quality realistic photos when dealing with pixel - aligned edge maps, but perform poorly when faced with highly deformed and sparse free - hand - drawn sketches. These methods usually assume that there is a pixel - level alignment between the input sketch and the target image, so the user's drawing skills directly affect the quality of the generated results. This makes it difficult for untrained amateurs to use existing systems to generate satisfactory photos.

To solve this problem, this paper proposes a new autoregressive sketch - to - photo generation model, aiming to generate highly realistic photos from highly abstract sketches. The model achieves this goal in the following ways:

1. **Decoupled encoder - decoder training paradigm**: The decoder is a StyleGAN pre - trained only on real photos, ensuring that the generated results always have a realistic quality.
2. **Autoregressive sketch mapper**: By training an autoregressive mapper to map the abstract sketch into the latent space of StyleGAN, the abstract gap between the sketch and the photo is dealt with.
3. **Fine - grained discriminative loss**: A fine - grained discriminative loss based on a pre - trained sketch - photo retrieval model is introduced to better capture the user's fine - grained intentions.
4. **Part - aware sketch enhancement strategy**: By rendering partial versions of the complete sketch and assigning corresponding latent vectors, the model's robustness to noise and partial sketches is improved.

These innovations enable the model to generate high - quality photos from sketches at different levels of abstraction and have good generalization ability. In addition, the model also shows its application potential in downstream tasks such as fine - grained sketch - based image retrieval (FG - SBIR) and precise semantic editing.