Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians

Kerui Ren,Lihan Jiang,Tao Lu,Mulin Yu,Linning Xu,Zhangkai Ni,Bo Dai

2024-10-17

Abstract:The recent 3D Gaussian splatting (3D-GS) has shown remarkable rendering fidelity and efficiency compared to NeRF-based neural scene representations. While demonstrating the potential for real-time rendering, 3D-GS encounters rendering bottlenecks in large scenes with complex details due to an excessive number of Gaussian primitives located within the viewing frustum. This limitation is particularly noticeable in zoom-out views and can lead to inconsistent rendering speeds in scenes with varying details. Moreover, it often struggles to capture the corresponding level of details at different scales with its heuristic density control operation. Inspired by the Level-of-Detail (LOD) techniques, we introduce Octree-GS, featuring an LOD-structured 3D Gaussian approach supporting level-of-detail decomposition for scene representation that contributes to the final rendering results. Our model dynamically selects the appropriate level from the set of multi-resolution anchor points, ensuring consistent rendering performance with adaptive LOD adjustments while maintaining high-fidelity rendering results.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in real - time rendering in large - scale scenes, the existing 3D Gaussian Splatting (3D - GS) method has problems of inefficiency and difficulty in detail capture due to a large number of Gaussian primitives. Specifically, 3D - GS performs poorly in scaled views because it renders all Gaussian primitives regardless of their projected sizes on the screen, which leads to inefficient use of model capacity and difficulty in capturing details at different scales. In addition, the distribution of Gaussian primitives in 3D - GS is inconsistent with the actual scene structure, resulting in reduced robustness when rendering views that are significantly different from the training set, and there are redundant and overlapping primitives in large - scale urban scenes, unable to efficiently represent scene details for real - time rendering.

To address these problems, the paper proposes Octree - GS, a method based on the Level of Detail (LOD) structure, which ensures consistent rendering performance by dynamically selecting the appropriate levels of multi - scale Gaussian primitives. The main contributions of Octree - GS include:

1. **Solve the LOD problem in Gaussian representation for the first time**: Through the explicit octree structure design, the dynamically adjusted acquisition of LOD is achieved, thus ensuring a consistent rendering speed.
2. **Develop a novel growth and pruning strategy**: Optimize LOD adaptability.
3. **Introduce a progressive training strategy**: Encourage a more reliable primitive distribution.
4. **The LOD strategy can be generalized to any Gaussian - based method**: Including explicit Gaussians and neural Gaussians.
5. **Achieve state - of - the - art rendering speed while maintaining superior rendering quality**: Especially in large - scale scenes and extreme - view sequences.

Through these improvements, Octree - GS not only reaches the state - of - the - art level in visual quality and rendering speed, but also performs particularly well when dealing with large - scale scenes.