Abstract:Objects obscured by occluders are considered lost in the images acquired by conventional camera systems, prohibiting both visualization and understanding of such hidden objects. Non-line-of-sight methods (NLOS) aim at recovering information about hidden scenes, which could help make medical imaging less invasive, improve the safety of autonomous vehicles, and potentially enable capturing unprecedented high-definition RGB-D data sets that include geometry beyond the directly visible parts. Recent NLOS methods have demonstrated scene recovery from time-resolved pulse-illuminated measurements encoding occluded objects as faint indirect reflections. Unfortunately, these systems are fundamentally limited by the quartic intensity fall-off for diffuse scenes. With laser illumination limited by eye-safety limits, recovery algorithms must tackle this challenge by incorporating scene priors. However, existing NLOS reconstruction algorithms do not facilitate learning scene priors. Even if they did, datasets that allow for such supervision do not exist, and successful encoder-decoder networks and generative adversarial networks fail for real-world NLOS data. In this work, we close this gap by learning hidden scene feature representations tailored to both reconstruction and recognition tasks such as classification or object detection, while still relying on physical models at the feature level. We overcome the lack of real training data with a generalizable architecture that can be trained in simulation. We learn the differentiable scene representation jointly with the reconstruction task using a differentiable transient renderer in the objective, and demonstrate that it generalizes to unseen classes and unseen real-world scenes , unlike existing encoder-decoder architectures and generative adversarial networks. The proposed method allows for end-to-end training for different NLOS tasks , such as image reconstruction, classification, and object detection, while being memory-efficient and running at real-time rates. We demonstrate hidden view synthesis, RGB-D reconstruction, classification, and object detection in the hidden scene in an end-to-end fashion.

Robust Speckle-Autocorrelation Non-Line-of-sight Imaging with Generative Adversarial Networks

Ghost Imaging with N-order Speckle Light

Dynamic-excitation-based Steady-State Non-Line-of-sight Imaging Via Multi-Branch Convolutional Neural Network

Deep Non-line-of-sight Imaging from Under-scanning Measurements.

Non-Line-of-Sight Imaging with Signal Superresolution Network

Learned feature embeddings for non-line-of-sight imaging and recognition

Towards Non-Line-of-Sight Photography

Recent Advances on Non-Line-of-Sight Imaging: Conventional Physical Models, Deep Learning, and New Scenes

Long-Range Non-Line-of-Sight Imaging Based on Projected Images from Multiple Light Fields

Non-line-of-sight Imaging with Arbitrary Illumination and Detection Pattern

Fast Non-Line-of-Sight Imaging With Hybrid Super-Resolution Network Over 18 m

Non-line-of-sight Sparse Aperture ISAR Imaging Via A Novel Detail-aware Regularization

Dynamic Non-Line-of-sight Imaging System Based on the Optimization of Point Spread Functions.

Passive Non-line-of-sight Imaging for Moving Targets with an Event Camera

Multi-scale Iterative Model-guided Unfolding Network for NLOS Reconstruction.

Plug-and-Play Algorithms for Dynamic Non-line-of-sight Imaging

Non-line-of-sight reconstruction via structure sparsity regularization

Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging

Model-Guided Iterative Diffusion Sampling for NLOS Reconstruction

45.2: High‐performance Non‐Line‐of‐Sight imaging based on deep learning

Non-line-of-sight imaging with adaptive artifact cancellation