Joint Material and Illumination Estimation from Photo Sets in the Wild

Tuanfeng Y. Wang,Tobias Ritschel,Nilov J. Mitra,Niloy J. Mitra
DOI: https://doi.org/10.1109/3dv.2018.00014
2018-09-01
Abstract:Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e., diffuse and specular) and illumination (i.e., environment maps). In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e., environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e., specular materials and environment illumination) for all the observed materials and environments. Technically, we enable this by a novel scalable formulation using parametric mixture models that allows for simultaneous estimation of all materials and illumination directly from a set of (uncontrolled) Internet images. At the core is an optimization that uses two neural networks trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study. Code and data are available on the project website11http://geometry.cs.ucl.ac.uk/projects/2018/joint-material-illumination-estimation. http://geometry.cs.ucl.ac.uk/projects/2018/joint-material-illumination-estimation
What problem does this paper attempt to address?