RayGauss: Volumetric Gaussian-Based Ray Casting for Photorealistic Novel View Synthesis

Hugo Blanc,Jean-Emmanuel Deschaud,Alexis Paljic

2024-08-06

Abstract:Differentiable volumetric rendering-based methods made significant progress in novel view synthesis. On one hand, innovative methods have replaced the Neural Radiance Fields (NeRF) network with locally parameterized structures, enabling high-quality renderings in a reasonable time. On the other hand, approaches have used differentiable splatting instead of NeRF's ray casting to optimize radiance fields rapidly using Gaussian kernels, allowing for fine adaptation to the scene. However, differentiable ray casting of irregularly spaced kernels has been scarcely explored, while splatting, despite enabling fast rendering times, is susceptible to clearly visible artifacts. Our work closes this gap by providing a physically consistent formulation of the emitted radiance c and density {\sigma}, decomposed with Gaussian functions associated with Spherical Gaussians/Harmonics for all-frequency colorimetric representation. We also introduce a method enabling differentiable ray casting of irregularly distributed Gaussians using an algorithm that integrates radiance fields slab by slab and leverages a BVH structure. This allows our approach to finely adapt to the scene while avoiding splatting artifacts. As a result, we achieve superior rendering quality compared to the state-of-the-art while maintaining reasonable training times and achieving inference speeds of 25 FPS on the Blender dataset. Project page with videos and code: <a class="link-external link-https" href="https://raygauss.github.io/" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The paper primarily aims to address several key issues in the field of Novel View Synthesis (NVS), particularly focusing on radiance field-based methods. Specifically, the research aims to:

1. **Improve rendering quality and efficiency**: By proposing a new method to represent the radiance field in a scene, achieving high-quality and efficient rendering effects. Existing methods, while making some progress in improving rendering speed, often sacrifice rendering quality.

2. **Overcome the limitations of different techniques**:
- For methods based on local parameter structures (such as voxel grids, tetrahedral meshes, etc.), these methods can achieve real-time rendering but have limitations in adapting to complex scene geometry details.
- For methods based on differentiable splatting, although they can quickly optimize the radiance field, they tend to produce noticeable artifacts during rendering, especially when deviating from the training viewpoints.

3. **Propose a new method with good physical consistency**: The paper introduces a method called RayGauss, which decomposes the radiance field into a set of non-uniformly distributed Gaussian basis functions, combined with Spherical Harmonics (SH) and Spherical Gaussians (SG) to represent color information at all frequencies. This method allows for finer adaptation to scene geometry while avoiding common artifacts in splatting methods.

4. **Achieve an efficient volumetric ray tracing algorithm**: To handle the non-uniformly distributed Gaussian basis functions, the paper proposes a ray tracing algorithm based on sequential integration and utilizes Bounding Volume Hierarchy (BVH) to accelerate the computation process. This method can achieve reasonable training time and real-time rendering speed while maintaining good rendering quality.

In summary, the main goal of this research is to develop a new method that can maintain high-quality rendering while achieving efficient computation, addressing the issues present in existing radiance field-based novel view synthesis methods.