Abstract:We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment. Multiview reconstruction of reflective objects is extremely challenging because specular reflections are view-dependent and thus violate the multiview consistency, which is the cornerstone for most multiview reconstruction methods. Recent neural rendering techniques can model the interaction between environment lights and the object surfaces to fit the view-dependent reflections, thus making it possible to reconstruct reflective objects from multiview images. However, accurately modeling environment lights in the neural rendering is intractable, especially when the geometry is unknown. Most existing neural rendering methods, which can model environment lights, only consider direct lights and rely on object masks to reconstruct objects with weak specular reflections. Therefore, these methods fail to reconstruct reflective objects, especially when the object mask is not available and the object is illuminated by indirect lights. We propose a two-step approach to tackle this problem. First, by applying the split-sum approximation and the integrated directional encoding to approximate the shading effects of both direct and indirect lights, we are able to accurately reconstruct the geometry of reflective objects without any object masks. Then, with the object geometry fixed, we use more accurate sampling to recover the environment lights and the BRDF of the object. Extensive experiments demonstrate that our method is capable of accurately reconstructing the geometry and the BRDF of reflective objects from only posed RGB images without knowing the environment lights and the object masks. Codes and datasets are available at https://github.com/liuyuan-pal/NeRO.

Neural Reflectance Capture in the View-Illumination Domain

Learning Efficient Illumination Multiplexing for Joint Capture of Reflectance and Shape

Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Efficient reflectance capture using an autoencoder.

NeuralTO: Neural Reconstruction and View Synthesis of Translucent Objects

Real-time Acquisition and Reconstruction of Dynamic Volumes with Neural Structured Illumination

Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection

S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint

A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields

NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images

Object-Based Illumination Estimation with Rendering-Aware Neural Networks

NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination

NeRFactor

NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild

IllumiNeRF: 3D Relighting without Inverse Rendering

Shadow-Aware Point-Based Neural Radiance Fields for High-Resolution Remote Sensing Novel View Synthesis

3D Reconstruction and New View Synthesis of Indoor Environments based on a Dual Neural Radiance Field

Neural 3D Reconstruction in the Wild

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

NeRFReN: Neural Radiance Fields with Reflections