Abstract:In 3D reconstruction, appearance is always overlooked compared to geometry and motion. The existing methods prefer texture-based shading, where the reconstructed appearance cannot perform re-lighting, thus weakens the authenticity, aesthetic and availability. Although excellent attempts were made in some works at lighting model-based shading in 3D reconstruction, still they fall short of differentiated rendering of materials. In this paper, we propose a method that can make an artistic effect on the recovery of illumination properties in 3D reconstruction from a single RGB-D image. Our method builds adaptive lighting models for different surfaces, which are used in re-lighting and recovering surface properties. We start with the establishment of coordinate system conversion tables for rapidly matching surface parameters and location. Then, a projection-based triangulation method is developed to generate polygonal mesh. After that, we propose a physically based lighting model with few variable parameters to recover surface illumination properties, in order to accurately describe the reflection of light from different materials. Furthermore, we develop an adaptive algorithm for generating parameters. Since the algorithm is based on the surface consistency of albedo over channels and training-less also, it is applicable to a surface of any shape and material, and the physically based lighting model brings a more vivid re-lighting. The obtained experimental results demonstrate that our method could achieve an excellent reconstructed appearance in both artistic view and verisimilitude.

Deep Inverse Rendering for Practical Object Appearance Scan with Uncalibrated Illumination

Calibration-free Deep Optics for Depth Estimation with Precise Simulation

NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images

Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images

IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination

Learning Efficient Illumination Multiplexing for Joint Capture of Reflectance and Shape

Object-Based Illumination Estimation with Rendering-Aware Neural Networks

Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting

Simultaneous Localization and Appearance Estimation with a Consumer RGB-D Camera.

Single Image Based Illumination Estimation for Lighting Virtual Object in Real Scene

WildLight: In-the-wild Inverse Rendering with a Flashlight

Physics-Based Appearance and Illumination Estimation from a Single Face Image

Lighting Virtual Objects in a Single Image Via Coarse Scene Understanding

Practical and Accurate Reconstruction of an Illuminant's Spectral Power Distribution for Inverse Rendering Pipelines

Adaptive Lighting Modeling in 3D Reconstruction with Illumination Properties Recovery

Neural Shading Fields for Efficient Facial Inverse Rendering

Eclipse: Disambiguating Illumination and Materials using Unintended Shadows

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image

Free-form Scanning of Non-Planar Appearance with Neural Trace Photography

Modeling Indirect Illumination for Inverse Rendering