Abstract:Geometry- and appearance-controlled full-body human image generation is an interesting but challenging task. Existing solutions are either unconditional or dependent on coarse conditions (e.g., pose, text), thus lacking explicit geometry and appearance control of body and garment. Sketching offers such editing ability and has been adopted in various sketch-based face generation and editing solutions. However, directly adapting sketch-based face generation to full-body generation often fails to produce high-fidelity and diverse results due to the high complexity and diversity in the pose, body shape, and garment shape and texture. Recent geometrically controllable diffusion-based methods mainly rely on prompts to generate appearance and it is hard to balance the realism and the faithfulness of their results to the sketch when the input is coarse. This work presents Sketch2Human, the first system for controllable full-body human image generation guided by a semantic sketch (for geometry control) and a reference image (for appearance control). Our solution is based on the latent space of StyleGAN-Human with inverted geometry and appearance latent codes as input. Specifically, we present a sketch encoder trained with a large synthetic dataset sampled from StyleGAN-Human's latent space and directly supervised by sketches rather than real images. Considering the entangled information of partial geometry and texture in StyleGAN-Human and the absence of disentangled datasets, we design a novel training scheme that creates geometry-preserved and appearance-transferred training data to tune a generator to achieve disentangled geometry and appearance control. Although our method is trained with synthetic data, it can handle hand-drawn sketches as well. Qualitative and quantitative evaluations demonstrate the superior performance of our method to state-of-the-art methods.

Geometry Sampling for 3D Face Generation Via DCGAN

SDF-based 3D Generative Adversarial Networks for Image and Geometry Generation

A survey of deep learning-based 3D shape generation

Geometry Guided Adversarial Facial Expression Synthesis

3D-GANTex: 3D Face Reconstruction with StyleGAN3-based Multi-View Images and 3DDFA based Mesh Generation

3DFaceGAN: Adversarial Nets for 3D Face Representation, Generation, and Translation

DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Geometry aware 3D generation from in-the-wild images in ImageNet

SDF‐StyleGAN: Implicit SDF‐Based StyleGAN for 3D Shape Generation

Single Image, Any Face: Generalisable 3D Face Generation

Geometry-Aware GAN for Face Attribute Transfer.

3D Face Reconstruction with Geometry Details from a Single Image

DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control

Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion

Measuring the Consistency and Diversity of 3D Face Generation.

G3FA: Geometry-guided GAN for Face Animation

Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Control

Geometry-Aware Face Completion and Editing

AG3D: Learning to Generate 3D Avatars from 2D Image Collections

DetailGen3D: Generative 3D Geometry Enhancement via Data-Dependent Flow

Face Sketch Synthesis via Semantic-Driven Generative Adversarial Network