Abstract:Objects obscured by occluders are considered lost in the images acquired by conventional camera systems, prohibiting both visualization and understanding of such hidden objects. Non-line-of-sight methods (NLOS) aim at recovering information about hidden scenes, which could help make medical imaging less invasive, improve the safety of autonomous vehicles, and potentially enable capturing unprecedented high-definition RGB-D data sets that include geometry beyond the directly visible parts. Recent NLOS methods have demonstrated scene recovery from time-resolved pulse-illuminated measurements encoding occluded objects as faint indirect reflections. Unfortunately, these systems are fundamentally limited by the quartic intensity fall-off for diffuse scenes. With laser illumination limited by eye-safety limits, recovery algorithms must tackle this challenge by incorporating scene priors. However, existing NLOS reconstruction algorithms do not facilitate learning scene priors. Even if they did, datasets that allow for such supervision do not exist, and successful encoder-decoder networks and generative adversarial networks fail for real-world NLOS data. In this work, we close this gap by learning hidden scene feature representations tailored to both reconstruction and recognition tasks such as classification or object detection, while still relying on physical models at the feature level. We overcome the lack of real training data with a generalizable architecture that can be trained in simulation. We learn the differentiable scene representation jointly with the reconstruction task using a differentiable transient renderer in the objective, and demonstrate that it generalizes to unseen classes and unseen real-world scenes , unlike existing encoder-decoder architectures and generative adversarial networks. The proposed method allows for end-to-end training for different NLOS tasks , such as image reconstruction, classification, and object detection, while being memory-efficient and running at real-time rates. We demonstrate hidden view synthesis, RGB-D reconstruction, classification, and object detection in the hidden scene in an end-to-end fashion.

45.2: High‐performance Non‐Line‐of‐Sight imaging based on deep learning

Dynamic-excitation-based Steady-State Non-Line-of-sight Imaging Via Multi-Branch Convolutional Neural Network

Long-Range Non-Line-of-Sight Imaging Based on Projected Images from Multiple Light Fields

Deep-Learning-Based Real-Time Passive Non-Line-of-Sight Imaging for Room-Scale Scenes

Fast Non-Line-of-Sight Imaging With Hybrid Super-Resolution Network Over 18 m

Non-Line-of-Sight Imaging with Signal Superresolution Network

Recent Advances on Non-Line-of-Sight Imaging: Conventional Physical Models, Deep Learning, and New Scenes

Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging

Real-time Non-Line-of-sight Computational Imaging Using Spectrum Filtering and Motion Compensation

Plug-and-Play Algorithms for Dynamic Non-line-of-sight Imaging

High-resolution non-line-of-sight imaging based on liquid crystal planar optical elements

PI-NLOS: polarized infrared non-line-of-sight imaging

Non–line-of-sight imaging over 1.43 km

Dynamic Non-Line-of-sight Imaging System Based on the Optimization of Point Spread Functions.

High-resolution non-confocal non-line-of-sight imaging based on spherical-slice transform from spatial and temporal frequency to space and time

An Efficient Way for Active None-Line-of-Sight: End-to-End Learned Compressed NLOS Imaging.

Accurate but Fragile Passive Non-Line-of-sight Recognition

Low-latency time-of-flight non-line-of-sight imaging at 5 frames per second

Learned feature embeddings for non-line-of-sight imaging and recognition

Passive None-line-of-sight imaging with arbitrary scene condition and detection pattern in small amount of prior data

Passive Non-Line-of-Sight Imaging Using Optimal Transport