RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting

Zhexi Peng,Tianjia Shao,Yong Liu,Jingke Zhou,Yin Yang,Jingdong Wang,Kun Zhou

DOI: https://doi.org/10.1145/3658233

2024-05-09

Abstract:We present Real-time Gaussian SLAM (RTG-SLAM), a real-time 3D reconstruction system with an RGBD camera for large-scale environments using Gaussian splatting. The system features a compact Gaussian representation and a highly efficient on-the-fly Gaussian optimization scheme. We force each Gaussian to be either opaque or nearly transparent, with the opaque ones fitting the surface and dominant colors, and transparent ones fitting residual colors. By rendering depth in a different way from color rendering, we let a single opaque Gaussian well fit a local surface region without the need of multiple overlapping Gaussians, hence largely reducing the memory and computation cost. For on-the-fly Gaussian optimization, we explicitly add Gaussians for three types of pixels per frame: newly observed, with large color errors, and with large depth errors. We also categorize all Gaussians into stable and unstable ones, where the stable Gaussians are expected to well fit previously observed RGBD images and otherwise unstable. We only optimize the unstable Gaussians and only render the pixels occupied by unstable Gaussians. In this way, both the number of Gaussians to be optimized and pixels to be rendered are largely reduced, and the optimization can be done in real time. We show real-time reconstructions of a variety of large scenes. Compared with the state-of-the-art NeRF-based RGBD SLAM, our system achieves comparable high-quality reconstruction but with around twice the speed and half the memory cost, and shows superior performance in the realism of novel view synthesis and camera tracking accuracy.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of real-time 3D reconstruction in large-scale environments. Specifically, it proposes a new system called RTG-SLAM (Real-time Gaussian SLAM), which utilizes Gaussian splatting technology to achieve efficient and high-quality large-scale 3D scene reconstruction.

The core contributions of the paper include:
1. **Compact Gaussian Representation**: By setting each Gaussian point to be either opaque or semi-transparent, a single opaque Gaussian point can fit local surface areas well, thereby reducing memory and computational costs.
2. **Efficient Online Gaussian Optimization Scheme**: By distinguishing between stable and unstable Gaussian points and optimizing only the unstable ones, the number of Gaussian points and rendering pixels that need optimization is greatly reduced, achieving real-time performance.
3. **Improved Depth Rendering Method**: Different methods are used to render depth, treating opaque Gaussian points as elliptical disks to better fit local surface areas and improve reconstruction accuracy.
4. **System Performance**: Compared to existing NeRF-based RGBD SLAM methods, RTG-SLAM is approximately twice as fast, consumes about half the memory, and excels in the realism of new view synthesis and camera tracking accuracy.

The paper demonstrates real-time reconstruction results of various real large-scale scenes (such as corridors, storage rooms, hotel rooms, homes, and offices) through experiments and compares them with existing methods, proving its superiority in terms of speed and memory efficiency.