Abstract:In this study, we introduce a methodology for human image animation by leveraging a 3D human parametric model within a latent diffusion framework to enhance shape alignment and motion guidance in curernt human generative techniques. The methodology utilizes the SMPL(Skinned Multi-Person Linear) model as the 3D human parametric model to establish a unified representation of body shape and pose. This facilitates the accurate capture of intricate human geometry and motion characteristics from source videos. Specifically, we incorporate rendered depth images, normal maps, and semantic maps obtained from SMPL sequences, alongside skeleton-based motion guidance, to enrich the conditions to the latent diffusion model with comprehensive 3D shape and detailed pose attributes. A multi-layer motion fusion module, integrating self-attention mechanisms, is employed to fuse the shape and motion latent representations in the spatial domain. By representing the 3D human parametric model as the motion guidance, we can perform parametric shape alignment of the human body between the reference image and the source video motion. Experimental evaluations conducted on benchmark datasets demonstrate the methodology's superior ability to generate high-quality human animations that accurately capture both pose and shape variations. Furthermore, our approach also exhibits superior generalization capabilities on the proposed in-the-wild dataset. Project page: <a class="link-external link-https" href="https://fudan-generative-vision.github.io/champ" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve the accuracy and consistency of shape alignment and motion guidance by using 3D human parameter models (such as the SMPL model) in human image animation generation. Specifically, the paper proposes a control method based on 3D parameter models, aiming to generate high - quality human image animations that can accurately capture the changes in posture and shape and have good generalization ability among different characters and scene dynamics. The paper points out that although existing GAN - based methods have achieved remarkable results in dynamic visual content generation, they often produce unrealistic visual artifacts and temporal inconsistencies when dealing with significant changes between human identities and scene dynamics. And although diffusion models can directly generate human animation videos under conditional guidance, they still have problems with insufficient handling of complex motions and details. Therefore, the method proposed in this paper aims to overcome the limitations of existing methods by introducing 3D human parameter models as motion guidance and optimizing the shape alignment and posture guidance mechanisms. Specifically, in terms of technology, this method is implemented through the following steps: 1. **Multi - layer motion conditions**: Extract 3D human parameter models (SMPL) from reference images and source videos, and generate multi - layer 3D - related information such as depth maps, normal maps, and semantic maps to provide rich conditional information. 2. **Parameterized shape alignment**: Adjust the shape parameters in the SMPL model to align the shape in the reference image with the motion sequence in the source video, thereby achieving pixel - level shape alignment. 3. **Multi - layer motion fusion**: Use the self - attention mechanism to fuse multi - layer feature embeddings and enhance the model's understanding of human postures and shapes. 4. **Network structure**: Combine the pre - trained denoising U - Net model and the reference encoder to ensure the consistency and coherence of the generated videos. Experimental results show that this method performs well on multiple benchmark datasets, can generate high - quality human animations, accurately capture the changes in posture and shape, and also shows strong generalization ability on in - the - wild datasets.

Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance

VividPose: Advancing Stable Video Diffusion for Realistic Human Image Animation

DisPose: Disentangling Pose Guidance for Controllable Human Image Animation

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation

MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance

HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation

Towards Accurate Markerless Human Shape and Pose Estimation over Time

High Quality Human Image Animation using Regional Supervision and Motion Blur Condition

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Free-viewpoint Human Animation with Pose-correlated Reference Selection

HSPACE: Synthetic Parametric Humans Animated in Complex Environments

Follow-Your-Pose v2: Multiple-Condition Guided Character Image Animation for Stable Pose Control

AMG: Avatar Motion Guided Video Generation

CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators

CHASE: 3D-Consistent Human Avatars with Sparse Inputs via Gaussian Splatting and Contrastive Learning

CapHuman: Capture Your Moments in Parallel Universes

AnchorCrafter: Animate CyberAnchors Saling Your Products via Human-Object Interacting Video Generation

FAAC: Facial Animation Generation with Anchor Frame and Conditional Control for Superior Fidelity and Editability

DreamDance: Animating Human Images by Enriching 3D Geometry Cues from 2D Poses

Deformable 3D Gaussian Splatting for Animatable Human Avatars