DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting

Shijie Zhou,Zhiwen Fan,Dejia Xu,Haoran Chang,Pradyumna Chari,Tejas Bharadwaj,Suya You,Zhangyang Wang,Achuta Kadambi

2024-07-25

Abstract:The increasing demand for virtual reality applications has highlighted the significance of crafting immersive 3D assets. We present a text-to-3D 360$^{\circ}$ scene generation pipeline that facilitates the creation of comprehensive 360$^{\circ}$ scenes for in-the-wild environments in a matter of minutes. Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement to create a high-quality and globally coherent panoramic image. This image acts as a preliminary "flat" (2D) scene representation. Subsequently, it is lifted into 3D Gaussians, employing splatting techniques to enable real-time exploration. To produce consistent 3D geometry, our pipeline constructs a spatially coherent structure by aligning the 2D monocular depth into a globally optimized point cloud. This point cloud serves as the initial state for the centroids of 3D Gaussians. In order to address invisible issues inherent in single-view inputs, we impose semantic and geometric constraints on both synthesized and input camera views as regularizations. These guide the optimization of Gaussians, aiding in the reconstruction of unseen regions. In summary, our method offers a globally consistent 3D scene within a 360$^{\circ}$ perspective, providing an enhanced immersive experience over existing techniques. Project website at: <a class="link-external link-http" href="http://dreamscene360.github.io/" rel="external noopener nofollow">this http URL</a>

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address the need for high-quality, immersive 3D scene generation in virtual reality (VR) applications. Specifically, the authors propose a method for generating 360-degree panoramic 3D scenes from text prompts, called **DreamScene360**. Current 3D scene generation methods face numerous challenges in creating globally consistent and detail-rich 3D scenes, especially when dealing with unconstrained outdoor environments. These issues include:

1. **Multi-view Inconsistency**: Existing 2D diffusion models often generate images that are inconsistent across multiple views, leading to incoherent geometric structures and textures in 3D scenes.
2. **Lack of Depth Information**: Images generated from single-view inputs lack depth information, making it difficult to create complete 3D geometric structures.
3. **Complex Prompt Engineering**: Generating high-quality 3D scenes usually requires complex prompt engineering, with users needing to make multiple attempts to achieve satisfactory results.
4. **Inability to Handle Unconstrained 360-degree Scenes**: Existing methods often produce distorted and incoherent structures when dealing with unconstrained 360-degree panoramic scenes.

### Solution

To overcome the above challenges, the DreamScene360 method adopts the following steps:

1. **Generate 360-degree Panoramic Images**:
- Utilize a pre-trained 2D diffusion model to generate a 360-degree panoramic image as an initial 2D representation of the 3D scene.
- Improve the quality of the generated image through a self-refinement mechanism, ensuring consistency with the text prompt.

2. **Initialize 3D Geometric Structure**:
- Use a monocular depth estimator to generate a depth map of the panoramic image, initializing a 3D point cloud.
- Align the point cloud by optimizing the geometric field, ensuring global consistency of the 3D structure.

3. **Optimize 3D Gaussian Distribution**:
- Convert the 2D panoramic image into a 3D Gaussian distribution, using a virtual camera to synthesize parallax and fill in occluded areas.
- Optimize the 3D Gaussian distribution through semantic and geometric regularization terms, ensuring semantic and geometric consistency in the generated 3D scene.

### Main Contributions

1. **Globally Consistent 3D Scene Generation**: DreamScene360 can generate 360-degree panoramic 3D scenes with global consistency, providing a more immersive experience.
2. **User-friendly Generation Process**: The self-refinement mechanism reduces the trial-and-error in prompt engineering, making the generation process more user-friendly and efficient.
3. **Handling Unconstrained 360-degree Scenes**: Capable of generating high-quality 3D scenes under unconstrained conditions, suitable for various indoor and outdoor environments.

### Experimental Results

The paper demonstrates the advantages of DreamScene360 in generating 360-degree panoramic 3D scenes through comparative experiments with existing methods such as LucidDreamer. The experimental results show that the scenes generated by DreamScene360 are more consistent in terms of semantics, style, and geometry, with no blind spots, providing a better immersive experience.