Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality

Hannah Schieber,Jacob Young,Tobias Langlotz,Stefanie Zollmann,Daniel Roth

2024-09-24

Abstract:Advancements in 3D rendering like Gaussian Splatting (GS) allow novel view synthesis and real-time rendering in virtual reality (VR). However, GS-created 3D environments are often difficult to edit. For scene enhancement or to incorporate 3D assets, segmenting Gaussians by class is essential. Existing segmentation approaches are typically limited to certain types of scenes, e.g., ''circular'' scenes, to determine clear object boundaries. However, this method is ineffective when removing large objects in non-''circling'' scenes such as large outdoor scenes. We propose Semantics-Controlled GS (SCGS), a segmentation-driven GS approach, enabling the separation of large scene parts in uncontrolled, natural environments. SCGS allows scene editing and the extraction of scene parts for VR. Additionally, we introduce a challenging outdoor dataset, overcoming the ''circling'' setup. We outperform the state-of-the-art in visual quality on our dataset and in segmentation quality on the 3D-OVS dataset. We conducted an exploratory user study, comparing a 360-video, plain GS, and SCGS in VR with a fixed viewpoint. In our subsequent main study, users were allowed to move freely, evaluating plain GS and SCGS. Our main study results show that participants clearly prefer SCGS over plain GS. We overall present an innovative approach that surpasses the state-of-the-art both technically and in user experience.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The main problem this paper attempts to address is the difficulty of editing existing Gaussian Splatting (GS) methods in outdoor scene reconstruction and rendering. Specifically:

1. **Editing Challenges**: The 3D environments created by existing GS methods are difficult to edit, especially when it is crucial to enhance the scene or integrate 3D assets by categorically segmenting Gaussian points.
2. **Scene Limitations**: Current segmentation methods are often limited to specific types of scenes (such as "ring" scenes), which perform poorly when dealing with large outdoor scenes.
3. **Non-"Ring" Scene Challenges**: For large outdoor scenes that are not "ring" shaped, removing large objects becomes very difficult because the feature similarity in these scenes makes segmentation more complex.

To address these issues, the authors propose a Semantic-Controlled GS (SCGS) method, which achieves the functionality of separating large scene parts in uncontrolled natural environments through a semantic segmentation-driven approach. Additionally, they introduce a challenging outdoor dataset and surpass the state-of-the-art in both visual quality and segmentation quality. Through user studies, they found that participants significantly preferred SCGS over the ordinary GS method. Overall, the paper provides an innovative approach that surpasses the existing level in both technology and user experience.