Abstract:Consistent human-centric image and video synthesis aims to generate images or videos with new poses while preserving appearance consistency with a given reference image, which is crucial for low-cost visual content creation. Recent advances based on diffusion models typically rely on separate networks for reference appearance feature extraction and target visual generation, leading to inconsistent domain gaps between references and targets. In this paper, we frame the task as a spatially-conditioned inpainting problem, where the target image is inpainted to maintain appearance consistency with the reference. This approach enables the reference features to guide the generation of pose-compliant targets within a unified denoising network, thereby mitigating domain gaps. Additionally, to better maintain the reference appearance information, we impose a causal feature interaction framework, in which reference features can only query from themselves, while target features can query appearance information from both the reference and the target. To further enhance computational efficiency and flexibility, in practical implementation, we decompose the spatially-conditioned generation process into two stages: reference appearance extraction and conditioned target generation. Both stages share a single denoising network, with interactions restricted to self-attention layers. This proposed method ensures flexible control over the appearance of generated human images and videos. By fine-tuning existing base diffusion models on human video data, our method demonstrates strong generalization to unseen human identities and poses without requiring additional per-instance fine-tuning. Experimental results validate the effectiveness of our approach, showing competitive performance compared to existing methods for consistent human image and video synthesis.

Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priors

Chasing Consistency in Text-to-3D Generation from a Single Image.

Consistent123: Improve Consistency for One Image to 3D Object Synthesis

One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior

Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models

ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors

Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior

Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization

Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model

Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle

CCD-3DR: Consistent Conditioning in Diffusion for Single-Image 3D Reconstruction.

Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video

Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy

EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior

Consistent Human Image and Video Generation with Spatially Conditioned Diffusion

Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding

ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models

SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D