3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes

Nicolas Moenne-Loccoz,Ashkan Mirzaei,Or Perel,Riccardo de Lutio,Janick Martinez Esturo,Gavriel State,Sanja Fidler,Nicholas Sharp,Zan Gojcic

2024-10-10

Abstract:Particle-based representations of radiance fields such as 3D Gaussian Splatting have found great success for reconstructing and re-rendering of complex scenes. Most existing methods render particles via rasterization, projecting them to screen space tiles for processing in a sorted order. This work instead considers ray tracing the particles, building a bounding volume hierarchy and casting a ray for each pixel using high-performance GPU ray tracing hardware. To efficiently handle large numbers of semi-transparent particles, we describe a specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order. The benefits of ray tracing are well-known in computer graphics: processing incoherent rays for secondary lighting effects such as shadows and reflections, rendering from highly-distorted cameras common in robotics, stochastically sampling rays, and more. With our renderer, this flexibility comes at little cost compared to rasterization. Experiments demonstrate the speed and accuracy of our approach, as well as several applications in computer graphics and vision. We further propose related improvements to the basic Gaussian representation, including a simple use of generalized kernel functions which significantly reduces particle hit counts.

Graphics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the limitations of particle-based scene representations (such as Gaussian particles) in rendering, particularly the compromises inherent in current rasterization techniques. Specifically, the goals of the paper include:

1. **Improving Rendering Efficiency**: By designing an efficient algorithm specifically for GPU-accelerated ray tracing to handle a large number of semi-transparent particles, enabling fast ray tracing of complex scenes.

2. **Enhancing Rendering Effects**: To enable ray tracing to efficiently simulate secondary light effects (such as reflection, refraction, shadows, etc.) and highly distorted camera effects (such as rolling shutter effects).

3. **Supporting Random Sampling**: Allowing the use of random sampling techniques during training, which is particularly important for computer vision tasks.

4. **Optimizing Particle Representation**: Proposing an improved Gaussian particle formula to reduce the number of collision detections and improve rendering efficiency.

5. **Providing Flexible Application Scenarios**: Demonstrating the advantages of ray tracing in various applications, such as depth of field, mirror effects, highly distorted cameras, rolling shutter effects, etc.

In summary, the core objective of the paper is to develop an efficient differentiable ray tracing algorithm for particle-based radiance field representations and to demonstrate its superior performance in various computer graphics and vision tasks.