Abstract:Objects obscured by occluders are considered lost in the images acquired by conventional camera systems, prohibiting both visualization and understanding of such hidden objects. Non-line-of-sight methods (NLOS) aim at recovering information about hidden scenes, which could help make medical imaging less invasive, improve the safety of autonomous vehicles, and potentially enable capturing unprecedented high-definition RGB-D data sets that include geometry beyond the directly visible parts. Recent NLOS methods have demonstrated scene recovery from time-resolved pulse-illuminated measurements encoding occluded objects as faint indirect reflections. Unfortunately, these systems are fundamentally limited by the quartic intensity fall-off for diffuse scenes. With laser illumination limited by eye-safety limits, recovery algorithms must tackle this challenge by incorporating scene priors. However, existing NLOS reconstruction algorithms do not facilitate learning scene priors. Even if they did, datasets that allow for such supervision do not exist, and successful encoder-decoder networks and generative adversarial networks fail for real-world NLOS data. In this work, we close this gap by learning hidden scene feature representations tailored to both reconstruction and recognition tasks such as classification or object detection, while still relying on physical models at the feature level. We overcome the lack of real training data with a generalizable architecture that can be trained in simulation. We learn the differentiable scene representation jointly with the reconstruction task using a differentiable transient renderer in the objective, and demonstrate that it generalizes to unseen classes and unseen real-world scenes , unlike existing encoder-decoder architectures and generative adversarial networks. The proposed method allows for end-to-end training for different NLOS tasks , such as image reconstruction, classification, and object detection, while being memory-efficient and running at real-time rates. We demonstrate hidden view synthesis, RGB-D reconstruction, classification, and object detection in the hidden scene in an end-to-end fashion.

Model-Guided Iterative Diffusion Sampling for NLOS Reconstruction

Applying Convolutional Neural Networks for the Source Reconstruction

Dynamic-excitation-based Steady-State Non-Line-of-sight Imaging Via Multi-Branch Convolutional Neural Network

Non-Line-of-Sight Imaging with Signal Superresolution Network

Curvature regularization for Non-line-of-sight Imaging from Under-sampled Data

Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging

A Reconstruction Method of Electromagnetic Scattering Target Based on Diffusion Model

Self-Calibrating, Fully Differentiable NLOS Inverse Rendering

Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning.

CT Reconstruction using Diffusion Posterior Sampling conditioned on a Nonlinear Measurement Model

Fast Non-Line-of-Sight Imaging With Hybrid Super-Resolution Network Over 18 m

Non-line-of-sight reconstruction via structure sparsity regularization

Non-line-of-Sight Imaging via Neural Transient Fields

Recent Advances on Non-Line-of-Sight Imaging: Conventional Physical Models, Deep Learning, and New Scenes

Learned feature embeddings for non-line-of-sight imaging and recognition

Fast Sampling generative model for Ultrasound image reconstruction

Diffusion Posterior Sampling for Nonlinear CT Reconstruction

Non-line-of-sight Imaging with Arbitrary Illumination and Detection Pattern

Long-Range Non-Line-of-Sight Imaging Based on Projected Images from Multiple Light Fields

Domain Reduction Strategy for Non Line of Sight Imaging

Plug-and-Play Algorithms for Dynamic Non-line-of-sight Imaging