Non-Lambertian photometric stereo via neural inverse rendering and interreflection model
Yuze Yang,Jiahang Liu,Yangyu Fu,Yue Ni
DOI: https://doi.org/10.1016/j.optlaseng.2024.108138
IF: 5.666
2024-03-08
Optics and Lasers in Engineering
Abstract:Photometric stereo (PS) is an important method for 3D fine reconstruction of object surfaces, and nonlinear mapping is the main method for solving surface normals in photometric stereo. The PS models are generally established under the assumption that the object has a consistent surface material and a purely convex surface. However, real-world objects often have surfaces with spatially varying materials (e.g., pigments) and global illumination effects (e.g., reflections and shadows due to complex shapes). The normals and details of these surfaces cannot be efficiently reconstructed because the basic assumptions of the nonlinear mapping normal solution are not satisfied. To address these issues, an inverse rendering neural framework PS-SINI is proposed in this paper through jointly optimizing surface normal by shadow rendering, parameterizing BRDF, and modeling interreflection. To eliminate photometric differences caused by material discontinuity, we decompose BRDF into diffuse reflection and specular reflection, and establish two models, PositonNet and SpecularNet, to estimate the diffuse reflection and specular reflection characteristics of each surface point respectively. To eliminate the effect of global illumination, we propose a DepthNet to generate depth map and then integrate ray tracing techniques to render the shadow map. To solve the detail loss in the concave shadow areas and the brightness loss in the convex surfaces caused by only using a single light source rendering, a neural inverse rendering photometric model with interreflection estimation is proposed by considering the secondary reflection effect of each microfacet. Different from the previous methods, our method is entirely self-supervised and requires neither synthetic training dataset nor ground-truth surface normal. Experimental results on several real-world public datasets demonstrate our method achieves superiority over state-of-the-art supervised or unsupervised approaches.
optics