Abstract:The rising demand for creating lifelike avatars in the digital realm has led to an increased need for generating high-quality human videos guided by textual descriptions and poses. We propose Dancing Avatar, designed to fabricate human motion videos driven by poses and textual cues. Our approach employs a pretrained T2I diffusion model to generate each video frame in an autoregressive fashion. The crux of innovation lies in our adept utilization of the T2I diffusion model for producing video frames successively while preserving contextual relevance. We surmount the hurdles posed by maintaining human character and clothing consistency across varying poses, along with upholding the background's continuity amidst diverse human movements. To ensure consistent human appearances across the entire video, we devise an intra-frame alignment module. This module assimilates text-guided synthesized human character knowledge into the pretrained T2I diffusion model, synergizing insights from ChatGPT. For preserving background continuity, we put forth a background alignment pipeline, amalgamating insights from segment anything and image inpainting techniques. Furthermore, we propose an inter-frame alignment module that draws inspiration from an auto-regressive pipeline to augment temporal consistency between adjacent frames, where the preceding frame guides the synthesis process of the current frame. Comparisons with state-of-the-art methods demonstrate that Dancing Avatar exhibits the capacity to generate human videos with markedly superior quality, both in terms of human and background fidelity, as well as temporal coherence compared to existing state-of-the-art approaches.

Text/Speech-Driven Full-Body Animation

Text-driven 3D Avatar Animation with Emotional and Expressive Behaviors.

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Text to Avatar in Multi-modal Human Computer Interface

Learning Speech-driven 3D Conversational Gestures from Video

Disentangled Clothed Avatar Generation from Text Descriptions

3D Visible Speech Animation Driven by Prosody Text

Video-driven state-aware facial animation

AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text

Text2Avatar: Text to 3D Human Avatar Generation with Codebook-Driven Body Controllable Attribute

Real-time Conversion from a Single 2D Face Image to a 3D Text-Driven Emotive Audio-Visual Avatar

Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Model

Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation

TALK-Act: Enhance Textural-Awareness for 2D Speaking Avatar Reenactment with Diffusion Model

SpeechAct: Towards Generating Whole-body Motion from Speech

Generating Holistic 3D Human Motion from Speech

3D Realistic Talking Face Co-Driven by Text and Speech

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

Towards Streaming Speech-to-Avatar Synthesis

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

Joint Audio-Video Driven Facial Animation