Abstract:We present a novel inverse rendering-based framework to estimate the 3D shape (per-pixel surface normals and depth) of objects and scenes from single-view polarization images, the problem popularly known as Shape from Polarization (SfP). The existing physics-based and learning-based methods for SfP perform under certain restrictions, i.e., (a) purely diffuse or purely specular reflections, which are seldom in the real surfaces, (b) availability of the ground truth surface normals for direct supervision that are hard to acquire and are limited by the scanner's resolution, and (c) known refractive index. To overcome these restrictions, we start by learning to separate the partially-polarized diffuse and specular reflection components, which we call reflectance cues, based on a modified polarization reflection model and then estimate shape under mixed polarization through an inverse-rendering based self-supervised deep learning framework called SS-SfP, guided by the polarization data and estimated reflectance cues. Furthermore, we also obtain the refractive index as a non-linear least squares solution. Through extensive quantitative and qualitative evaluation, we establish the efficacy of the proposed framework over simple single-object scenes from DeepSfP dataset and complex in-the-wild scenes from SPW dataset in an entirely self-supervised setting. To the best of our knowledge, this is the first learning-based approach to address SfP under mixed polarization in a completely self-supervised framework.

What problem does this paper attempt to address?

The paper aims to address the problem of estimating the 3D shape (i.e., pixel-level surface normals and depth) of objects or scenes from single-view polarized images, a problem known as "Shape from Polarization" (SfP). Existing physics-based and learning-based methods have certain limitations when dealing with SfP, such as: 1. **Single Reflection Type**: Assuming pure diffuse reflection or pure specular reflection, while actual surfaces usually have mixed reflection characteristics. 2. **Direct Supervision Requirement**: Requiring real surface normals as direct supervision information, which is difficult to obtain in reality and limited by scanner resolution. 3. **Known Refractive Index**: Requiring the refractive index of the material to be known, otherwise the estimated shape will be affected by refractive distortion. To address these limitations, the authors propose a new self-supervised deep learning framework based on neural inverse rendering (referred to as SS-SfP) for shape estimation under mixed polarization conditions. Specifically, the method first separates the partially polarized diffuse and specular reflection components through a modified polarization reflection model and uses these reflection cues to estimate the shape. Additionally, the refractive index is obtained by solving a nonlinear least squares problem. Through extensive quantitative and qualitative evaluations, the framework demonstrates superior performance on the DeepSfP dataset and the SPW dataset, especially in a fully self-supervised setting. To the best of the authors' knowledge, this is the first time the SfP problem under mixed polarization conditions has been addressed in a fully self-supervised framework.

SS-SfP:Neural Inverse Rendering for Self Supervised Shape from (Mixed) Polarization

Shape from Polarization with Distant Lighting Estimation.

Deep Polarization Cues for Single-shot Shape and Subsurface Scattering Estimation

Deep Shape from Polarization

Predicting Polarization Beyond Semantics for Wearable Robotics.

Transparent Shape from a Single View Polarization Image

S2P3: Self-Supervised Polarimetric Pose Prediction

Polarimetric Inverse Rendering for Transparent Shapes Reconstruction

3D Human Shape Reconstruction from a Polarization Image

S P : Self-Supervised Polarimetric Pose Prediction

SPIDeRS: Structured Polarization for Invisible Depth and Reflectance Sensing

Shape from polarization for featureless and specular objects

PISR: Polarimetric Neural Implicit Surface Reconstruction for Textureless and Specular Objects

PANDORA: Polarization-Aided Neural Decomposition Of Radiance

Recent Progress in Shape from Polarization

A Geometric Model for Polarization Imaging on Projective Cameras

NeISF: Neural Incident Stokes Field for Geometry and Material Estimation

Near-infrared monocular 3D computational polarization imaging of surfaces exhibiting nonuniform reflectance

DeepPS2: Revisiting Photometric Stereo Using Two Differently Illuminated Images

Human Pose and Shape Estimation from Single Polarization Images

Robust Depth Enhancement via Polarization Prompt Fusion Tuning