Abstract:Photometric stereo (PS) is an important method for 3D fine reconstruction of object surfaces, and nonlinear mapping is the main method for solving surface normals in photometric stereo. The PS models are generally established under the assumption that the object has a consistent surface material and a purely convex surface. However, real-world objects often have surfaces with spatially varying materials (e.g., pigments) and global illumination effects (e.g., reflections and shadows due to complex shapes). The normals and details of these surfaces cannot be efficiently reconstructed because the basic assumptions of the nonlinear mapping normal solution are not satisfied. To address these issues, an inverse rendering neural framework PS-SINI is proposed in this paper through jointly optimizing surface normal by shadow rendering, parameterizing BRDF, and modeling interreflection. To eliminate photometric differences caused by material discontinuity, we decompose BRDF into diffuse reflection and specular reflection, and establish two models, PositonNet and SpecularNet, to estimate the diffuse reflection and specular reflection characteristics of each surface point respectively. To eliminate the effect of global illumination, we propose a DepthNet to generate depth map and then integrate ray tracing techniques to render the shadow map. To solve the detail loss in the concave shadow areas and the brightness loss in the convex surfaces caused by only using a single light source rendering, a neural inverse rendering photometric model with interreflection estimation is proposed by considering the secondary reflection effect of each microfacet. Different from the previous methods, our method is entirely self-supervised and requires neither synthetic training dataset nor ground-truth surface normal. Experimental results on several real-world public datasets demonstrate our method achieves superiority over state-of-the-art supervised or unsupervised approaches.

Non-Lambertian Multispectral Photometric Stereo Via Spectral Reflectance Decomposition

Multispectral Photometric Stereo for Spatially-Varying Spectral Reflectances

A Non-Lambertian Photometric Stereo under Perspective Projection

Learning Inter- and Intra-frame Representations for Non-Lambertian Photometric Stereo

Non-Lambertian photometric stereo by color and noise analysis

NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

Numerical Reflectance Compensation for Non-Lambertian Photometric Stereo

Summary Study of Data-Driven Photometric Stereo Methods

Non-Lambertian photometric stereo via neural inverse rendering and interreflection model

Deep Photometric Stereo Networks for Determining Surface Normal and Reflectances.

Multiscale Convolutional Fusion Network for Non-Lambertian Photometric Stereo

Deep Photometric Stereo for Non-Lambertian Surfaces

Deep Photometric Stereo Network

Non-Lambertian Photometric Stereo Network Based on Inverse Reflectance Model with Collocated Light.

Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo

Physics-Free Spectrally Multiplexed Photometric Stereo under Unknown Spectral Composition

MS-PS: A Multi-Scale Network for Photometric Stereo With a New Comprehensive Training Dataset

Non-Lambertian Photometric Stereo Vision Based On Inverse Reflectance Model

NormAttention-PSN: A High-frequency Region Enhanced Photometric Stereo Network with Normalized Attention

Multi-view photometric stereo of non-Lambertian surface under general illuminations

A Novel Learning Based Non-Lambertian Photometric Stereo Method for Pixel-Level Normal Reconstruction of Polished Surfaces