RRM: Relightable assets using Radiance guided Material extraction

Diego Gomez,Julien Philip,Adrien Kaiser,Élie Michel

2024-07-09

Abstract:Synthesizing NeRFs under arbitrary lighting has become a seminal problem in the last few years. Recent efforts tackle the problem via the extraction of physically-based parameters that can then be rendered under arbitrary lighting, but they are limited in the range of scenes they can handle, usually mishandling glossy scenes. We propose RRM, a method that can extract the materials, geometry, and environment lighting of a scene even in the presence of highly reflective objects. Our method consists of a physically-aware radiance field representation that informs physically-based parameters, and an expressive environment light structure based on a Laplacian Pyramid. We demonstrate that our contributions outperform the state-of-the-art on parameter retrieval tasks, leading to high-fidelity relighting and novel view synthesis on surfacic scenes.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The main objective of this paper is to address the problem of synthesizing NeRF (Neural Radiance Fields) under arbitrary lighting conditions. Specifically:

1. **Physical Parameter Extraction**:
- A method (RRM) is proposed that can extract materials, geometry, and environmental lighting from a scene, even in the presence of highly reflective objects.

2. **High-Fidelity Relighting and Novel View Synthesis**:
- Through an environment lighting representation method based on the Laplacian pyramid and a physically-aware radiance field representation, this method can achieve high-fidelity relighting and novel view synthesis on surface scenes.

3. **Handling High-Gloss Scenes**:
- Compared to existing methods (such as NMF and TensoIR), RRM is more robust in handling glossy materials and can better manage self-reflections.

4. **End-to-End Optimization**:
- This method is an end-to-end optimizable architecture, including a physically-aware radiance module and a physics-based rendering module, which can work together to further eliminate information ambiguity in complex scenes.

In summary, the paper aims to develop a new method for extracting physical parameters from a set of input images and re-rendering the scene under new lighting conditions, with particularly higher performance in handling glossy and rough surfaces.