Abstract:The combination of range sensors with color cameras can be very useful for robot navigation, semantic perception, manipulation, and telepresence. Several methods of combining range- and color-data have been investigated and successfully used in various robotic applications. Most of these systems suffer from the problems of noise in the range-data and resolution mismatch between the range sensor and the color cameras, since the resolution of current range sensors is much less than the resolution of color cameras. High-resolution depth maps can be obtained using stereo matching, but this often fails to construct accurate depth maps of weakly/repetitively textured scenes, or if the scene exhibits complex self-occlusions. Range sensors provide coarse depth information regardless of presence/absence of texture. The use of a calibrated system, composed of a time-of-flight (TOF) camera and of a stereoscopic camera pair, allows data fusion thus overcoming the weaknesses of both individual sensors. We propose a novel TOF-stereo fusion method based on an efficient seed-growing algorithm which uses the TOF data projected onto the stereo image pair as an initial set of correspondences. These initial "seeds" are then propagated based on a Bayesian model which combines an image similarity score with rough depth priors computed from the low-resolution range data. The overall result is a dense and accurate depth map at the resolution of the color cameras at hand. We show that the proposed algorithm outperforms 2D image-based stereo algorithms and that the results are of higher resolution than off-the-shelf color-range sensors, e.g., Kinect. Moreover, the algorithm potentially exhibits real-time performance on a single CPU.

ToF depth map guided passive stereo based on KNN searching

K-nearest Neighborhood Based Integration of Time-of-flight Cameras and Passive Stereo for High-Accuracy Depth Maps.

K-nearestneighborhood Basedintegrationoftime-of-flight Camerasandpassivestereoforhigh-Accuracydepthmaps ∗

Reliable Fusion of ToF and Stereo Data Based on Joint Depth Filter

High quality depth maps from stereo matching and ToF camera.

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Sparse LIDAR Measurement Fusion with Joint Updating Cost for Fast Stereo Matching

Improveing Depth Estimation Through Fusion of Stereo and TOF

Real-time depth completion based on LiDAR-stereo for autonomous driving

Research on TOF camera-based guided binocular fusion technology

Expanding Sparse LiDAR Depth and Guiding Stereo Matching for Robust Dense Depth Estimation

Robust and accurate depth estimation by fusing LiDAR and Stereo

High-Resolution Depth Maps Based on TOF-Stereo Fusion

Iterative Error Removal for Time-of-Flight Depth Imaging

Variational Fusion of Time-of-Flight and Stereo Data for Depth Estimation Using Edge-Selective Joint Filtering

FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras

3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization

3D Reconstruction Using 3D Registration-Based ToF-Stereo Fusion

DELTAR: Depth Estimation from a Light-Weight ToF Sensor and RGB Image

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

High-Quality Depth Recovery Via Interactive Multi-view Stereo