NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections

Dor Verbin,Pratul P. Srinivasan,Peter Hedman,Ben Mildenhall,Benjamin Attal,Richard Szeliski,Jonathan T. Barron

2024-05-24

Abstract:Neural Radiance Fields (NeRFs) typically struggle to reconstruct and render highly specular objects, whose appearance varies quickly with changes in viewpoint. Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. Moreover, these techniques rely on large computationally-expensive neural networks to model outgoing radiance, which severely limits optimization and rendering speed. We address these issues with an approach based on ray tracing: instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts reflection rays from these points and traces them through the NeRF representation to render feature vectors which are decoded into color using a small inexpensive network. We demonstrate that our model outperforms prior methods for view synthesis of scenes containing shiny objects, and that it is the only existing NeRF method that can synthesize photorealistic specular appearance and reflections in real-world scenes, while requiring comparable optimization time to current state-of-the-art view synthesis models.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The main goal of this paper is to improve the performance of Neural Radiance Fields (NeRF) in rendering specular objects, specifically addressing the following two major issues:

1. **Synthesis of High-Quality Reflections**: Existing NeRF methods perform poorly when handling specular reflections of near-field scene content, typically only accurately rendering specular reflections caused by distant environmental lighting. The method proposed in this paper is capable of generating high-quality specular reflections in real scenes, and is particularly effective for near-field content.

2. **Computational Efficiency**: Traditional methods rely on large neural networks to represent view-dependent radiance, which limits optimization and rendering speed. The method proposed in this paper introduces ray tracing technology, reducing the computational load required to represent highly detailed view-dependent functions at each scene point, thereby improving computational efficiency.

In summary, this paper aims to propose a new method that not only enhances the quality of NeRF in rendering specular objects but also achieves higher computational efficiency while maintaining optimization times comparable to existing state-of-the-art view synthesis models.