Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Model

Bosheng Qin,Wentao Ye,Qifan Yu,Siliang Tang,Yueting Zhuang

DOI: https://doi.org/10.48550/arXiv.2308.07749

2023-08-15

Computer Vision and Pattern Recognition

Abstract:The rising demand for creating lifelike avatars in the digital realm has led to an increased need for generating high-quality human videos guided by textual descriptions and poses. We propose Dancing Avatar, designed to fabricate human motion videos driven by poses and textual cues. Our approach employs a pretrained T2I diffusion model to generate each video frame in an autoregressive fashion. The crux of innovation lies in our adept utilization of the T2I diffusion model for producing video frames successively while preserving contextual relevance. We surmount the hurdles posed by maintaining human character and clothing consistency across varying poses, along with upholding the background's continuity amidst diverse human movements. To ensure consistent human appearances across the entire video, we devise an intra-frame alignment module. This module assimilates text-guided synthesized human character knowledge into the pretrained T2I diffusion model, synergizing insights from ChatGPT. For preserving background continuity, we put forth a background alignment pipeline, amalgamating insights from segment anything and image inpainting techniques. Furthermore, we propose an inter-frame alignment module that draws inspiration from an auto-regressive pipeline to augment temporal consistency between adjacent frames, where the preceding frame guides the synthesis process of the current frame. Comparisons with state-of-the-art methods demonstrate that Dancing Avatar exhibits the capacity to generate human videos with markedly superior quality, both in terms of human and background fidelity, as well as temporal coherence compared to existing state-of-the-art approaches.

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address the need for generating realistic digital avatars, specifically by synthesizing high-quality human motion videos guided by text descriptions and poses. Specifically, the paper proposes a new method called "Dancing Avatar," which utilizes a pre-trained text-to-image (T2I) diffusion model to generate continuous and spatiotemporally consistent human motion videos.

#### Main Contributions:
1. **Proposing Dancing Avatar**: An innovative method that uses a T2I diffusion model to synthesize human motion videos guided by text and poses.
2. **Frame Alignment Module and Background Alignment Pipeline**: Ensures consistency in characters and backgrounds between adjacent frames.
3. **Experimental Results**: Compared to existing state-of-the-art methods, Dancing Avatar can generate significantly higher quality videos with better input condition consistency and higher inter-frame coherence.

#### Specific Challenges Addressed:
1. **Maintaining Character Appearance Consistency**: Ensuring consistency in clothing and facial features of the character under different pose conditions.
2. **Maintaining Background Consistency**: Ensuring background consistency across different human motions.

By introducing an intra-frame alignment module, an inter-frame alignment module, and a background alignment pipeline, the paper addresses these challenges and demonstrates significant improvements in visual quality and coherence.