DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation

Jiaxiang Tang,Jiawei Ren,Hang Zhou,Ziwei Liu,Gang Zeng

2024-03-29

Abstract:Recent advances in 3D content creation mostly leverage optimization-based 3D generation via score distillation sampling (SDS). Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks. To further enhance the texture quality and facilitate downstream applications, we introduce an efficient algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details. Extensive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach. Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-view image, achieving approximately 10 times acceleration compared to existing methods.

Computer Science

What problem does this paper attempt to address?

The paper proposes a new framework called DreamGaussian to speed up optimization-based 3D content creation, particularly for tasks involving image and text to 3D conversion. The current methods are limited by their slow sample-wise optimization. DreamGaussian addresses this issue by utilizing a 3D Gaussian layer model and associated mesh extraction with texture refinement. Compared to neural radiance fields, DreamGaussian achieves faster convergence for 3D generation tasks through progressive Gaussian densification. Additionally, it introduces an efficient algorithm for the conversion from 3D Gaussian to texture meshes and a UV space texture refinement stage to enhance texture quality. Experimental results demonstrate that DreamGaussian can generate high-quality texture meshes in a matter of minutes, approximately 10 times faster than existing methods.