PoseCrafter: One-Shot Personalized Video Synthesis Following Flexible Pose Control

Yong Zhong,Min Zhao,Zebin You,Xiaofeng Yu,Changwang Zhang,Chongxuan Li

2024-07-18

Abstract:In this paper, we introduce PoseCrafter, a one-shot method for personalized video generation following the control of flexible poses. Built upon Stable Diffusion and ControlNet, we carefully design an inference process to produce high-quality videos without the corresponding ground-truth frames. First, we select an appropriate reference frame from the training video and invert it to initialize all latent variables for generation. Then, we insert the corresponding training pose into the target pose sequences to enhance faithfulness through a trained temporal attention module. Furthermore, to alleviate the face and hand degradation resulting from discrepancies between poses of training videos and inference poses, we implement simple latent editing through an affine transformation matrix involving facial and hand landmarks. Extensive experiments on several datasets demonstrate that PoseCrafter achieves superior results to baselines pre-trained on a vast collection of videos under 8 commonly used metrics. Besides, PoseCrafter can follow poses from different individuals or artificial edits and simultaneously retain the human identity in an open-domain training video. Our project page is available at <a class="link-external link-https" href="https://ml-gsai.github.io/PoseCrafter-demo/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to solve several key challenges in personalized video generation, especially when it is required to generate high - quality videos based on flexible pose control. Specifically, the paper focuses on the following aspects:

1. **Large - scale data requirements**: Existing pose - guided image - to - video methods usually require a large amount of high - quality human video data for training, which are often difficult to obtain and costly to process.
2. **Dependence on real frames**: Many existing video editing methods rely heavily on real video frames corresponding to the target pose, which is a major obstacle in generating new content.
3. **Facial and hand degradation**: Due to the differences between training videos and inference poses, facial and hand details often degrade in the generated videos.

To overcome these problems, the paper proposes **PoseCrafter**, a one - shot - based method that can generate high - quality personalized videos without relying on a large amount of training data and can accurately follow flexible pose control. The main contributions of PoseCrafter include:

- **Reference frame selection and insertion**: Improve the fidelity of the generated video by selecting an appropriate reference frame and inserting its pose into the target pose sequence.
- **Simple latent variable editing**: Edit the latent variables through an affine transformation matrix to improve the degradation problem of facial and hand details.
- **Efficient data and computation utilization**: Compared with existing image - to - video methods, PoseCrafter is more efficient in data and computation.

Through these techniques, PoseCrafter can achieve better results than existing baseline methods on multiple datasets and can maintain the identity characteristics of the character under different pose controls.