A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights

Wentao Lei,Jinting Wang,Fengji Ma,Guanjie Huang,Li Liu

2024-07-11

Abstract:Human video generation is a dynamic and rapidly evolving task that aims to synthesize 2D human body video sequences with generative models given control conditions such as text, audio, and pose. With the potential for wide-ranging applications in film, gaming, and virtual communication, the ability to generate natural and realistic human video is critical. Recent advancements in generative models have laid a solid foundation for the growing interest in this area. Despite the significant progress, the task of human video generation remains challenging due to the consistency of characters, the complexity of human motion, and difficulties in their relationship with the environment. This survey provides a comprehensive review of the current state of human video generation, marking, to the best of our knowledge, the first extensive literature review in this domain. We start with an introduction to the fundamentals of human video generation and the evolution of generative models that have facilitated the field's growth. We then examine the main methods employed for three key sub-tasks within human video generation: text-driven, audio-driven, and pose-driven motion generation. These areas are explored concerning the conditions that guide the generation process. Furthermore, we offer a collection of the most commonly utilized datasets and the evaluation metrics that are crucial in assessing the quality and realism of generated videos. The survey concludes with a discussion of the current challenges in the field and suggests possible directions for future research. The goal of this survey is to offer the research community a clear and holistic view of the advancements in human video generation, highlighting the milestones achieved and the challenges that lie ahead.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the Paper Aims to Address

This paper aims to provide a comprehensive review of the current state of human video generation, systematically analyzing the challenges, methods, and future research directions in this field. Specifically:

1. **Defining the Task Scope**: The paper first clarifies the boundaries of the human video generation task, which involves using generative models to synthesize natural and realistic 2D human video sequences based on control conditions such as text, audio, or poses.

2. **Method Classification**: Existing methods are categorized into three major types: text-based methods, audio-based methods, and pose-based methods. The paper discusses in detail how these methods guide video generation during the process.

3. **Challenges and Datasets**: The paper delves into the challenges faced in human video generation and compiles a large number of related datasets, evaluation metrics, and commercial projects. This helps readers choose suitable foundational models or solutions for specific applications and provides valuable insights for improving existing methods.

4. **Future Research Directions**: Through literature review and in-depth analysis, the paper identifies potential future research directions in human motion generation.

In summary, the goal of this review is to provide the research community with a clear and comprehensive view of the progress in the field of human video generation, highlighting the milestones achieved and the challenges faced.