Abstract:Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e., diffuse and specular) and illumination (i.e., environment maps). In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e., environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e., specular materials and environment illumination) for all the observed materials and environments. Technically, we enable this by a novel scalable formulation using parametric mixture models that allows for simultaneous estimation of all materials and illumination directly from a set of (uncontrolled) Internet images. At the core is an optimization that uses two neural networks trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study. Code and data are available on the project website11http://geometry.cs.ucl.ac.uk/projects/2018/joint-material-illumination-estimation. http://geometry.cs.ucl.ac.uk/projects/2018/joint-material-illumination-estimation

Learning Efficient Illumination Multiplexing for Joint Capture of Reflectance and Shape

Efficient reflectance capture using an autoencoder.

Neural Reflectance Capture in the View-Illumination Domain

A Unified Spatial-Angular Structured Light for Single-View Acquisition of Shape and Reflectance

Learning Inter- and Intra-frame Representations for Non-Lambertian Photometric Stereo

Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images

Learning optimal lighting patterns for efficient SVBRDF acquisition

Joint Material and Illumination Estimation from Photo Sets in the Wild

Two-Shot Spatially-Varying BRDF and Shape Estimation

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Real-time Acquisition and Reconstruction of Dynamic Volumes with Neural Structured Illumination

Collaborative Reflectance-And-Illumination Learning For High-Efficient Low-Light Image Enhancement.

Learning Illumination from Diverse Portraits

Deep Inverse Rendering for Practical Object Appearance Scan with Uncalibrated Illumination

Shape and Material Capture at Home

Learning Efficient Photometric Feature Transform for Multi-view Stereo

Self-adaptive Single and Multi-illuminant Estimation Framework based on Deep Learning

Free-form Scanning of Non-Planar Appearance with Neural Trace Photography

Efficient Reflectance Capture with a Deep Gated Mixture-of-Experts

SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild

Spatiotemporally Consistent HDR Indoor Lighting Estimation