Abstract:Physics-based inverse rendering aims to jointly optimize shape, materials, and lighting from captured 2D images. Here lighting is an important part of achieving faithful light transport simulation. While the environment map is commonly used as the lighting model in inverse rendering, we show that its distant lighting assumption leads to spatial invariant lighting, which can be an inaccurate approximation in real-world inverse rendering. We propose to use NeRF as a spatially varying environment lighting model and build an inverse rendering pipeline using NeRF as the non-distant environment emitter. By comparing our method with the environment map on real and synthetic datasets, we show that our NeRF-based emitter models the scene lighting more accurately and leads to more accurate inverse rendering. Project page and video:

What problem does this paper attempt to address?

This paper mainly discusses how to simulate the non-distant lighting environment more accurately in physics-based inverse rendering. Traditional inverse rendering usually approximates lighting using environment maps, but this method introduces errors when dealing with non-distant light sources, leading to shadowing and distorted shape reconstruction results. The paper proposes to use Neural Radiance Field (NeRF) as a non-distant environment emitter to improve the accuracy of lighting modeling. NeRF is a 3D radiance field model that can better represent the spatially varying lighting distribution, especially suitable for handling scenes with non-distant light sources. By comparing the performance of using NeRF and environment maps, the paper demonstrates that NeRF achieves superior results on both real and synthetic datasets, enabling higher-quality inverse rendering. In terms of methodology, the paper constructs a hybrid scene model where objects are represented by surface and material, while the surrounding environmental lighting is modeled by NeRF. By introducing emitter importance sampling specifically for NeRF, the variance of the rendering equation is reduced, and a pipeline for simultaneous shape, material, and NeRF lighting reconstruction in inverse rendering is built. The related work section mentions the application of NeRF in novel view synthesis and the development of differentiable rendering techniques. The paper points out that although existing inverse rendering methods usually assume simple lighting models, this limits their applicability in real-world scenes. In contrast, the proposed NeRF emitter can handle non-distant lighting more accurately, thereby improving the performance of inverse rendering. In conclusion, the paper addresses how to use NeRF to simulate non-distant lighting more accurately in physics-based inverse rendering, thereby improving the reconstruction quality of shape, material, and lighting parameters.

NeRF as Non-Distant Environment Emitter in Physics-based Inverse Rendering

NeRF As a Non-Distant Environment Emitter in Physics-based Inverse Rendering

IllumiNeRF: 3D Relighting without Inverse Rendering

A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields

NeRFactor

NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination

NePF: Neural Photon Field for Single-Stage Inverse Rendering

ESR-NeRF: Emissive Source Reconstruction Using LDR Multi-view Images

IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis

Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes

Neural Fields Meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes

NeRFReN: Neural Radiance Fields with Reflections

NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields

Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption

NeRF synthesis with shading guidance

NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects

Prior-Driven NeRF: Prior Guided Rendering

SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field

NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

Enhance-NeRF: Multiple Performance Evaluation for Neural Radiance Fields