3D Gaussian Splatting for Real-Time Radiance Field Rendering

Bernhard Kerbl,Georgios Kopanas,Thomas Leimkühler,George Drettakis

2023-08-08

Abstract:Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

Graphics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of real-time high-quality Radiance Field Rendering, particularly the challenges faced when synthesizing new viewpoint images from multiple photos or videos in real scenes.

Specifically, the goals of the paper can be summarized as follows:

1. **Achieve real-time rendering**: Propose a new solution to achieve high-quality real-time rendering, especially at a rendering rate of over 30 frames per second (fps) at 1080p resolution, addressing the issues of long training times and slow rendering speeds in current methods.

2. **Optimize training time and quality**: Shorten the training time while ensuring rendering quality, making it comparable to the fastest current radiance field methods, and match or even surpass the quality of the best implicit radiance field methods.

3. **Balance speed and quality**: Overcome the problem of existing fast methods sacrificing quality for speed by proposing a new method that ensures efficient training speed while maintaining high rendering quality.

To address the above issues, the paper proposes three key technical points:

1. **3D Gaussian distribution representation**: Utilize 3D Gaussian distribution to represent the scene, inheriting the advantages of continuous volumetric radiance fields while avoiding unnecessary empty space computations.

2. **Optimization and density control of 3D Gaussian distribution parameters**: Achieve precise scene representation by interleaving the optimization of the position, anisotropic covariance matrix, and other attributes of the 3D Gaussian distribution, and by adding or removing 3D Gaussian distributions as needed.

3. **Fast visibility-aware rendering algorithm**: Develop a fast rendering algorithm that supports anisotropic scattering, which not only accelerates the training process but also makes real-time rendering possible.

In summary, the main contribution of this paper is the proposal of a new method that combines 3D Gaussian distribution representation and efficient rendering techniques, finding an ideal balance between training time and rendering quality, and achieving high-quality real-time radiance field rendering.