Abstract:Geometry- and appearance-controlled full-body human image generation is an interesting but challenging task. Existing solutions are either unconditional or dependent on coarse conditions (e.g., pose, text), thus lacking explicit geometry and appearance control of body and garment. Sketching offers such editing ability and has been adopted in various sketch-based face generation and editing solutions. However, directly adapting sketch-based face generation to full-body generation often fails to produce high-fidelity and diverse results due to the high complexity and diversity in the pose, body shape, and garment shape and texture. Recent geometrically controllable diffusion-based methods mainly rely on prompts to generate appearance and it is hard to balance the realism and the faithfulness of their results to the sketch when the input is coarse. This work presents Sketch2Human, the first system for controllable full-body human image generation guided by a semantic sketch (for geometry control) and a reference image (for appearance control). Our solution is based on the latent space of StyleGAN-Human with inverted geometry and appearance latent codes as input. Specifically, we present a sketch encoder trained with a large synthetic dataset sampled from StyleGAN-Human's latent space and directly supervised by sketches rather than real images. Considering the entangled information of partial geometry and texture in StyleGAN-Human and the absence of disentangled datasets, we design a novel training scheme that creates geometry-preserved and appearance-transferred training data to tune a generator to achieve disentangled geometry and appearance control. Although our method is trained with synthetic data, it can handle hand-drawn sketches as well. Qualitative and quantitative evaluations demonstrate the superior performance of our method to state-of-the-art methods.

High-Fidelity Portrait Editing Via Exploring Differentiable Guided Sketches from the Latent Space.

DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN

Flexible Portrait Image Editing with Fine-Grained Control

DiffSketching: Sketch Control Image Synthesis with Diffusion Models

Face photo-sketch portraits transformation via generation pipeline

Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches

SketchFFusion: Sketch-guided image editing with diffusion model

Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Control

Controllable Sketch-to-Image Translation for Robust Face Synthesis

Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale

The Devil is in the Details: StyleFeatureEditor for Detail-Rich StyleGAN Inversion and High Quality Image Editing

CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing

Mask-Guided Portrait Editing With Conditional GANs

Quality Metric Guided Portrait Line Drawing Generation from Unpaired Training Data

Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation

Towards Criminal Sketching with Generative Adversarial Network

Sketch Your Own GAN

HCGAN: hierarchical contrast generative adversarial network for unpaired sketch face synthesis

GuidedStyle: Attribute knowledge guided style manipulation for semantic face editing

Face Sketch Synthesis via Semantic-Driven Generative Adversarial Network