Abstract:3D-shape reconstruction in extreme environments, such as low illumination or scattering condition, has been an open problem and intensively researched. Active stereo is one of potential solution for such environments for its robustness and high accuracy. However, active stereo systems usually consist of specialized system configurations with complicated algorithms, which narrow their application. In this paper, we propose Neural Signed Distance Field for active stereo systems to enable implicit correspondence search and triangulation in generalized Structured Light. With our technique, textureless or equivalent surfaces by low light condition are successfully reconstructed even with a small number of captured images. Experiments were conducted to confirm that the proposed method could achieve state-of-the-art reconstruction quality under such severe condition. We also demonstrated that the proposed method worked in an underwater scenario.

What problem does this paper attempt to address?

This paper attempts to solve the problem of 3D shape reconstruction in extreme environments (such as low - light or scattering conditions). Specifically, the authors propose a new method - ActiveNeuS, which aims to improve the performance of active stereo systems in these extreme environments by introducing the neural signed distance field (Neural SDF) and the volume rendering technique of pattern projection. ### Main problems 1. **3D reconstruction in low - light or texture - less areas**: Traditional methods perform poorly in low - light or texture - less areas because it is difficult to find the correspondence between images under these conditions. 2. **Reducing the number of required input images**: Traditional active stereo systems usually require a large number of input images to ensure the reconstruction quality, which limits their application range. 3. **Adapting to the illumination changes in the wild environment**: During the actual scanning process, the illumination conditions may change significantly, which poses a challenge to the robustness of reconstruction. ### Solutions - **Implicit correspondence search and triangulation**: By using Neural SDF, this method can achieve implicit correspondence search and triangulation in the generalized structured light (SL) framework, thus reducing the need for a large number of input images. - **Illumination estimation**: Introduce illumination estimation techniques to adapt to the illumination changes in the wild environment and improve the robustness of reconstruction. - **Single - scan reconstruction**: Even with only one input image, this method can achieve accurate 3D reconstruction. ### Method overview 1. **System configuration**: It includes a camera and multiple projectors, and the internal and external parameters of all devices are accurately calibrated. 2. **Pipeline design**: - Randomly sample the rays emitted from the optical center of the camera. - Sample 3D points at regular or weighted intervals between the near clipping plane and the far clipping plane. - Pass the 3D points to the SDF network and the color network to obtain the density and diffuse color. - Reproject the 3D points onto the projector pattern and calculate the projected color. - Use volume rendering to generate an image and calculate the photometric loss between it and the real image. - Use the Adam optimizer to update the network parameters to minimize the photometric loss. ### Experimental results - **Synthetic data experiments**: Under different illumination conditions, ActiveNeuS is superior to NeuS in reconstruction accuracy, especially in low - light conditions. - **Real - data experiments**: In the underwater environment, ActiveNeuS can achieve dense and accurate reconstruction, especially showing obvious advantages in texture - less areas. ### Conclusion The ActiveNeuS method proposed in this paper performs well in 3D shape reconstruction in extreme environments (such as low - light or few input images), verifying its advantages over the traditional Neural SDF. In addition, this method also shows the ability of illumination estimation in the wild environment and the ability of single - scan reconstruction. ### Formula examples - **Volume rendering formula**: \[ C=\sum_{i = 1}^{n}\left(\prod_{j = 1}^{i - 1}(1-\alpha(P_j))\right)\alpha(P_i)c(P_i,v) \] where \(\alpha(P_i)\) is the opacity of the 3D point \(P_i\), \(c(P_i,v)\) is the color of the point \(P_i\) observed from the direction \(v\), and \(C\) is the final rendered ray color. - **Illumination mixing formula**: \[ c(P_i,v)=D(P_i,v)+\sum_{k = 1}^{N_p}(i_rD(P_i,v)+i_b)Q_k(P_i) \] where \(D(P_i,v)\) is the diffuse color of the point \(P_i\), \(Q_k(P_i)\) is the color projected by the \(k\)-th projector, and \(i_r\) and \(i_b\) are the reflection coefficient and the bias coefficient respectively. Through these innovations, ActiveNeuS is for extreme

ActiveNeuS: Neural Signed Distance Fields for Active Stereo

Depth Reconstruction with Neural Signed Distance Fields in Structured Light Systems

Neural Active Structure-from-Motion in Dark and Textureless Environment

Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo

SDFMAP: Neural Signed Distance Fields for Mapping and Positioning in Real-Time

Robust active stereo vision using Kullback-Leibler divergence

Sparse Views, Near Light: A Practical Paradigm for Uncalibrated Point-light Photometric Stereo

High-Quality Depth Recovery Via Interactive Multi-view Stereo

Learning Inter- and Intra-frame Representations for Non-Lambertian Photometric Stereo

Active stereo vision three-dimensional reconstruction by RGB dot pattern projection and ray intersection

Neural Field-Based Space Target 3D Reconstruction with Predicted Depth Priors

Non-learning Stereo-aided Depth Completion under Mis-projection via Selective Stereo Matching

Connecting The Dots: Learning Representations For Active Monocular Depth Estimation

ResDepth: A Deep Residual Prior For 3D Reconstruction From High-resolution Satellite Images

Dense Photometric Stereo: A Markov Random Field Approach

Edge-preserving Near-light Photometric Stereo with Neural Surfaces

Active Stereo Without Pattern Projector

A Learned Stereo Depth System for Robotic Manipulation in Homes

Learning Pseudo Front Depth for 2D Forward-Looking Sonar-based Multi-view Stereo

ActiveZero++: Mixed Domain Learning Stereo and Confidence-based Depth Completion with Zero Annotation

NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction With Implicit Neural Representations