GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality

Taoran Yi,Jiemin Fang,Zanwei Zhou,Junjie Wang,Guanjun Wu,Lingxi Xie,Xiaopeng Zhang,Wenyu Liu,Xinggang Wang,Qi Tian

2024-06-27

Abstract:Recently, 3D Gaussian splatting (3D-GS) has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. However, the generated assets have not achieved the same quality as those in reconstruction tasks. We observe that Gaussians tend to grow without control as the generation process may cause indeterminacy. Aiming at highly enhancing the generation quality, we propose a novel framework named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process. Along different stages of our framework, both the geometry and appearance can be enriched progressively. The final output asset is constructed with 3D Gaussians bound to mesh, which shows significantly enhanced details and quality compared with previous methods. Notably, the generated asset can also be seamlessly integrated into downstream manipulation pipelines, e.g. animation, composition, and simulation etc., greatly promoting its potential in wide applications. Demos are available at <a class="link-external link-https" href="https://taoranyi.com/gaussiandreamerpro/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

This paper mainly focuses on the problem of generating high-quality 3D Gaussian assets from text. Although current methods are capable of generating 3D Gaussian assets, their quality cannot match the performance in reconstruction tasks. The author found that during the generation process, the Gaussian assets grow uncontrollably, resulting in loss of details and surface blurriness. To address this issue, they propose a new framework called GaussianDreamerPro.

The core idea of GaussianDreamerPro is to bind the Gaussian assets to reasonable geometric structures, allowing them to develop in an orderly manner throughout the generation process. By gradually enriching the geometric and appearance details in three stages, the final generated assets are composed of 3D Gaussian assets bound to the mesh. These assets have significantly enhanced details and quality, and can seamlessly integrate into downstream manipulation pipelines such as animation, composition, and simulation.

The paper also compares with existing methods, pointing out that GaussianDreamerPro achieves a significant improvement in the quality of generated 3D assets compared to previous work. It is also compatible with other 3D generation methods and can further enhance the quality of assets generated by methods like DreamCraft3D.

Additionally, the paper introduces related work including text-based 3D generation models, 3D representation methods, etc. It provides a detailed description of the workflow and technical details of GaussianDreamerPro, including two stages: basic 3D asset generation and optimization of Gaussian assets through geometric constraints for quality enhancement. Finally, the paper presents experimental results and user studies, demonstrating the effectiveness and advantages of this method.