Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering

Linjie Lyu,Ayush Tewari,Marc Habermann,Shunsuke Saito,Michael Zollhöfer,Thomas Leimkühler,Christian Theobalt

DOI: https://doi.org/10.1145/3618357

2023-09-30

Abstract:Inverse rendering, the process of inferring scene properties from images, is a challenging inverse problem. The task is ill-posed, as many different scene configurations can give rise to the same image. Most existing solutions incorporate priors into the inverse-rendering pipeline to encourage plausible solutions, but they do not consider the inherent ambiguities and the multi-modal distribution of possible decompositions. In this work, we propose a novel scheme that integrates a denoising diffusion probabilistic model pre-trained on natural illumination maps into an optimization framework involving a differentiable path tracer. The proposed method allows sampling from combinations of illumination and spatially-varying surface materials that are, both, natural and explain the image observations. We further conduct an extensive comparative study of different priors on illumination used in previous work on inverse rendering. Our method excels in recovering materials and producing highly realistic and diverse environment map samples that faithfully explain the illumination of the input images.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The paper attempts to address the ambiguity problem present in the inverse rendering process. Specifically, the authors propose a novel method that utilizes a pre-trained Denoising Diffusion Probabilistic Model (DDPM) to handle the inherent multi-solution and uncertainty when inferring scene attributes (such as lighting and material) from images. This method can generate samples of lighting and material combinations that are both natural and explanatory of the input image, thereby improving the quality and diversity of inverse rendering results.

Through this method, the paper addresses the following main issues:
1. **Inverse Rendering under Multimodal Distribution**: Existing inverse rendering methods often converge to a local optimum and fail to fully explore all possible solutions. The proposed method can sample from a multimodal posterior distribution, generating diverse lighting and material combinations.
2. **Estimation of Natural Environment Lighting**: The proposed method can not only generate high-quality environment lighting maps but also edit and relight these lighting maps.
3. **Improving the Robustness of Inverse Rendering**: By combining DDPM and a differentiable path tracer, this method ensures that the generated results are natural while making them more consistent with the observation data of the input image.

In summary, the paper aims to address the inherent ambiguity in inverse rendering through a novel diffusion posterior sampling method, achieving high-quality and diverse inverse rendering results.