A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields

Yiyu Zhuang,Qi Zhang,Xuan Wang,Hao Zhu,Ying Feng,Xiaoyu Li,Ying Shan,Xun Cao
DOI: https://doi.org/10.1609/aaai.v38i7.28618
2024-01-01
Abstract:Recent advances in implicit neural representation have demonstrated the ability to recover detailed geometry and material from multi-view images. However, the use of simplified lighting models such as environment maps to represent non-distant illumination, or using a network to fit indirect light modeling without a solid basis, can lead to an undesirable decomposition between lighting and material. To address this, we propose a fully differentiable framework named Neural Illumination Fields (NeIF) that uses radiance fields as a lighting model to handle complex lighting in a physically based way. Together with integral lobe encoding for roughness-adaptive specular lobe and leveraging the pre-convolved background for accurate decomposition, the proposed method represents a significant step towards integrating physically based rendering into the NeRF representation. The experiments demonstrate the superior performance of novel-view rendering compared to previous works, and the capability to re-render objects under arbitrary NeRF-style environments opens up exciting possibilities for bridging the gap between virtual and real-world scenes.
What problem does this paper attempt to address?