BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion

Yonghao Yu,Shunan Zhu,Huai Qin,Haorui Li

DOI: https://doi.org/10.24963/ijcai.2024/598

2024-09-18

Abstract:Witnessing the evolution of text-to-image diffusion models, significant strides have been made in text-to-3D generation. Currently, two primary paradigms dominate the field of text-to-3D: the feed-forward generation solutions, capable of swiftly producing 3D assets but often yielding coarse results, and the Score Distillation Sampling (SDS) based solutions, known for generating high-fidelity 3D assets albeit at a slower pace. The synergistic integration of these methods holds substantial promise for advancing 3D generation techniques. In this paper, we present BoostDream, a highly efficient plug-and-play 3D refining method designed to transform coarse 3D assets into high-quality. The BoostDream framework comprises three distinct processes: (1) We introduce 3D model distillation that fits differentiable representations from the 3D assets obtained through feed-forward generation. (2) A novel multi-view SDS loss is designed, which utilizes a multi-view aware 2D diffusion model to refine the 3D assets. (3) We propose to use prompt and multi-view consistent normal maps as guidance in refinement.Our extensive experiment is conducted on different differentiable 3D representations, revealing that BoostDream excels in generating high-quality 3D assets rapidly, overcoming the Janus problem compared to conventional SDS-based methods. This breakthrough signifies a substantial advancement in both the efficiency and quality of 3D generation processes.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to efficiently generate high - quality 3D assets in the field of text - to - 3D generation. Currently, there are two main methods for generating 3D assets: one is the feed - forward generation method, which can quickly generate 3D assets, but the generated results are usually rather rough; the other is the method based on Score Distillation Sampling (SDS), which can generate high - fidelity 3D assets, but is slow and prone to the so - called "Janus problem" (i.e., the multi - head problem, referring to the phenomenon that 3D models generated from different viewing angles are inconsistent).

The paper proposes a new method - BoostDream, aiming to combine the advantages of these two methods to achieve efficient and high - quality 3D asset generation. Specifically, BoostDream achieves this goal through the following three stages:

1. **3D Model Distillation**: Convert the explicitly represented 3D assets obtained through the feed - forward generation method into a differentiable 3D representation form, making it trainable.
2. **Multi - view SDS Loss Design**: Utilize a multi - view - aware 2D diffusion model to refine 3D assets and solve the Janus problem.
3. **Use Prompts and Multi - view - consistent Normal Maps as Guidance**: During the refinement process, use text prompts and multi - view - consistent normal maps to guide the generation process, ensuring that the generated 3D assets are improved in both detail and quality.

Through these techniques, BoostDream can significantly improve the quality of 3D assets while maintaining the generation speed, thus solving the trade - off problem between efficiency and quality in existing methods.