Abstract:The rising demand for creating lifelike avatars in the digital realm has led to an increased need for generating high-quality human videos guided by textual descriptions and poses. We propose Dancing Avatar, designed to fabricate human motion videos driven by poses and textual cues. Our approach employs a pretrained T2I diffusion model to generate each video frame in an autoregressive fashion. The crux of innovation lies in our adept utilization of the T2I diffusion model for producing video frames successively while preserving contextual relevance. We surmount the hurdles posed by maintaining human character and clothing consistency across varying poses, along with upholding the background's continuity amidst diverse human movements. To ensure consistent human appearances across the entire video, we devise an intra-frame alignment module. This module assimilates text-guided synthesized human character knowledge into the pretrained T2I diffusion model, synergizing insights from ChatGPT. For preserving background continuity, we put forth a background alignment pipeline, amalgamating insights from segment anything and image inpainting techniques. Furthermore, we propose an inter-frame alignment module that draws inspiration from an auto-regressive pipeline to augment temporal consistency between adjacent frames, where the preceding frame guides the synthesis process of the current frame. Comparisons with state-of-the-art methods demonstrate that Dancing Avatar exhibits the capacity to generate human videos with markedly superior quality, both in terms of human and background fidelity, as well as temporal coherence compared to existing state-of-the-art approaches.

A Video-driven Approach to 3D Human Animation Synthesis

Sketch Based Multi-source 3D Animation Transfer

Video Based Human Animation Technique

Video-based Characters

3D Human Motion Synthesis Based on Nonlinear Manifold Learning

Action2video: Generating Videos of Human 3D Actions

Natural Video-based Human Motion Capture

Video-driven state-aware facial animation

Editing and Synthesis of 3D Facial Animation by Motion Propagation and Isomap Analysis

Automatic Generation of Human Animation Based on Motion Programming.

Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes

Automatic Generation of Human Animation Based on Motion Programming: Virtual Humans and Social Agents

Video Motion Capture in VBA—Video-based Animation

3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data

Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Model

Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis

AMG: Avatar Motion Guided Video Generation

SPA: Sparse Photorealistic Animation Using a Single RGB-D Camera

Depth-Varying human video sprite synthesis

A Piece-Wise Learning Approach To 3d Facial Animation

Generating 3D Virtual Human Animation Based on Facial Expression and Human Posture Captured by Dual Cameras