Progress and Prospects in 3D Generative AI: A Technical Overview including 3D human

Song Bai,Jie Li

2024-01-05

Abstract:While AI-generated text and 2D images continue to expand its territory, 3D generation has gradually emerged as a trend that cannot be ignored. Since the year 2023 an abundant amount of research papers has emerged in the domain of 3D generation. This growth encompasses not just the creation of 3D objects, but also the rapid development of 3D character and motion generation. Several key factors contribute to this progress. The enhanced fidelity in stable diffusion, coupled with control methods that ensure multi-view consistency, and realistic human models like SMPL-X, contribute synergistically to the production of 3D models with remarkable consistency and near-realistic appearances. The advancements in neural network-based 3D storing and rendering models, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have accelerated the efficiency and realism of neural rendered models. Furthermore, the multimodality capabilities of large language models have enabled language inputs to transcend into human motion outputs. This paper aims to provide a comprehensive overview and summary of the relevant papers published mostly during the latter half year of 2023. It will begin by discussing the AI generated object models in 3D, followed by the generated 3D human models, and finally, the generated 3D human motions, culminating in a conclusive summary and a vision for the future.

Artificial Intelligence,Graphics

What problem does this paper attempt to address?

The paper primarily explores the latest advancements and technological trends in Generative AI for 3D content generation. Specifically, the paper aims to review the following aspects:

1. **3D Object Generation**:
- The paper discusses various techniques for generating high-quality 3D models, including point cloud-based methods, waveform domain diffusion models, etc.
- It mentions specific methods such as MVDream and SweetDreamer, which generate high-quality 3D models from multi-view images and ensure model consistency.

2. **Human 3D Model Generation**:
- It introduces the development of human modeling technologies, particularly the application of SMPL and SMPL-X models, which significantly enhance the realism of human models.
- Several methods for generating high-precision human models are discussed, such as DreamWaltz and HumanNorm, which generate realistic human models through iterative optimization.

3. **3D Scene Generation**:
- The paper explores methods for generating 3D scenes from a single panoramic image, such as PERF, which can handle large occluded areas and generate complete 3D scenes.
- It mentions several techniques for generating complex 3D scenes, such as SceneDreamer, which can generate infinitely expanding 3D landscapes from a set of 2D images.

4. **Human Motion Synthesis**:
- Various techniques for generating human motions are analyzed, such as DIMOS and Story2Motion, which can generate continuous human motion sequences based on textual descriptions.
- The application of large language models in understanding the relationship between body movements and text is emphasized, making the generated motions more natural and smooth.

In summary, the paper provides a comprehensive overview of the latest research advancements in 3D generation technologies since the second half of 2023, covering various generation techniques from 3D objects to human models to complex scenes, and looks forward to future development directions.