NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields

Xiaoxue Chen,Junchen Liu,Hao Zhao,Guyue Zhou,Ya-Qin Zhang

2023-09-23

Abstract:Neural radiance fields (NeRF) have revolutionized the field of image-based view synthesis. However, NeRF uses straight rays and fails to deal with complicated light path changes caused by refraction and reflection. This prevents NeRF from successfully synthesizing transparent or specular objects, which are ubiquitous in real-world robotics and A/VR applications. In this paper, we introduce the refractive-reflective field. Taking the object silhouette as input, we first utilize marching tetrahedra with a progressive encoding to reconstruct the geometry of non-Lambertian objects and then model refraction and reflection effects of the object in a unified framework using Fresnel terms. Meanwhile, to achieve efficient and effective anti-aliasing, we propose a virtual cone supersampling technique. We benchmark our method on different shapes, backgrounds and Fresnel terms on both real-world and synthetic datasets. We also qualitatively and quantitatively benchmark the rendering results of various editing applications, including material editing, object replacement/insertion, and environment illumination estimation. Codes and data are publicly available at <a class="link-external link-https" href="https://github.com/dawning77/NeRRF" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper aims to address the complex optical phenomena of transparent and specular objects in 3D reconstruction and novel view synthesis. Specifically:

1. **Existing Issues**:
- Current Neural Radiance Field (NeRF) methods use straight light rays and cannot handle complex light path changes caused by refraction and reflection.
- This results in NeRF performing poorly when synthesizing transparent or specular objects.

2. **Objectives**:
- Propose a new method—Neural Refraction Reflection Field (NeRRF)—to model scenes containing non-Lambertian objects (such as glass, mirrors, etc.).
- Achieve high-quality novel view synthesis by reconstructing the 3D geometry of these objects and learning environmental lighting from refractive or reflective surfaces.

3. **Main Contributions**:
- Utilize a hybrid representation to reconstruct the 3D shape of non-Lambertian objects solely from object contours and use a progressive encoding strategy to accurately estimate object geometry.
- Explicitly model light path changes caused by refraction and reflection, and uniformly handle refraction and reflection based on the Fresnel equations.
- Propose an efficient anti-aliasing module that uses virtual cones for supersampling to reduce high-frequency noise.
- Construct a diverse benchmark dataset and demonstrate the superiority of NeRRF in various applications, including environmental lighting estimation, material editing, relighting, etc.

Through the above methods, NeRRF can effectively separate the geometric structure and appearance of transparent or specular objects and support various augmented reality (AR) and virtual reality (VR) applications.