Abstract:The emergence of diffusion models has enabled the generation of diverse high-quality images solely from text, prompting subsequent efforts to enhance the controllability of these models. Despite the improvement in controllability, pose control remains limited to specific objects (e.g., humans) or poses (e.g., frontal view) due to the fact that pose is generally controlled via camera parameters (e.g., rotation angle) or keypoints (e.g., eyes, nose). Specifically, camera parameters-conditional pose control models generate unrealistic images depending on the object, owing to the small size of 3D datasets for training. Also, keypoint-based approaches encounter challenges in acquiring reliable keypoints for various objects (e.g., church) or poses (e.g., back view). To address these limitations, we propose depth-based pose control, as depth maps are easily obtainable from a single depth estimation model regardless of objects and poses, unlike camera parameters and keypoints. However, depth-based pose control confronts issues of shape dependency, as depth maps influence not only the pose but also the shape of the generated images. To tackle this issue, we propose Skip-and-Play (SnP), designed via analysis of the impact of three components of depth-conditional ControlNet on the pose and the shape of the generated images. To be specific, based on the analysis, we selectively skip parts of the components to mitigate shape dependency on the depth map while preserving the pose. Through various experiments, we demonstrate the superiority of SnP over baselines and showcase the ability of SnP to generate images of diverse objects and poses. Remarkably, SnP exhibits the ability to generate images even when the objects in the condition (e.g., a horse) and the prompt (e.g., a hedgehog) differ from each other.

Controllable Person Image Synthesis with Pose-Constrained Latent Diffusion

Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis

Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models

Di^2Pose: Discrete Diffusion Model for Occluded 3D Human Pose Estimation

Correspondence Learning for Controllable Person Image Generation

Pose- and Attribute-consistent Person Image Synthesis

Novel 3D-Aware Composition Images Synthesis for Object Display with Diffusion Model.

Person Image Synthesis in Arbitrary 3D Poses Based on Part Affinity Fields.

LaCon: Late-Constraint Diffusion for Steerable Guided Image Synthesis

Skip-and-Play: Depth-Driven Pose-Preserved Image Generation for Any Objects

PersonaCraft: Personalized Full-Body Image Synthesis for Multiple Identities from Single References Using 3D-Model-Conditioned Diffusion

Controlling Human Shape and Pose in Text-to-Image Diffusion Models via Domain Adaptation

MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion

DiffPose: Toward More Reliable 3D Pose Estimation

Synthesizing Moving People with 3D Control

Controllable Person Image Synthesis with Spatially-Adaptive Warped Normalization

TCAN: Animating Human Images with Temporally Consistent Pose Guidance using Diffusion Models

Adaptive Multi-Modal Control of Digital Human Hand Synthesis Using a Region-Aware Cycle Loss

From Text to Pose to Image: Improving Diffusion Model Control and Quality

Diffusion-HPC: Generating Synthetic Images with Realistic Humans

PoseCrafter: One-Shot Personalized Video Synthesis Following Flexible Pose Control