Abstract:The combination of range sensors with color cameras can be very useful for robot navigation, semantic perception, manipulation, and telepresence. Several methods of combining range- and color-data have been investigated and successfully used in various robotic applications. Most of these systems suffer from the problems of noise in the range-data and resolution mismatch between the range sensor and the color cameras, since the resolution of current range sensors is much less than the resolution of color cameras. High-resolution depth maps can be obtained using stereo matching, but this often fails to construct accurate depth maps of weakly/repetitively textured scenes, or if the scene exhibits complex self-occlusions. Range sensors provide coarse depth information regardless of presence/absence of texture. The use of a calibrated system, composed of a time-of-flight (TOF) camera and of a stereoscopic camera pair, allows data fusion thus overcoming the weaknesses of both individual sensors. We propose a novel TOF-stereo fusion method based on an efficient seed-growing algorithm which uses the TOF data projected onto the stereo image pair as an initial set of correspondences. These initial "seeds" are then propagated based on a Bayesian model which combines an image similarity score with rough depth priors computed from the low-resolution range data. The overall result is a dense and accurate depth map at the resolution of the color cameras at hand. We show that the proposed algorithm outperforms 2D image-based stereo algorithms and that the results are of higher resolution than off-the-shelf color-range sensors, e.g., Kinect. Moreover, the algorithm potentially exhibits real-time performance on a single CPU.

Variational Fusion of Time-of-Flight and Stereo Data for Depth Estimation Using Edge-Selective Joint Filtering

Reliable Fusion of ToF and Stereo Data Based on Joint Depth Filter

K-nearest Neighborhood Based Integration of Time-of-flight Cameras and Passive Stereo for High-Accuracy Depth Maps.

High quality depth maps from stereo matching and ToF camera.

K-nearestneighborhood Basedintegrationoftime-of-flight Camerasandpassivestereoforhigh-Accuracydepthmaps ∗

Improveing Depth Estimation Through Fusion of Stereo and TOF

High-Resolution Depth Maps Based on TOF-Stereo Fusion

High-Quality Depth Recovery Via Interactive Multi-view Stereo

Sparse LIDAR Measurement Fusion with Joint Updating Cost for Fast Stereo Matching

Expanding Sparse LiDAR Depth and Guiding Stereo Matching for Robust Dense Depth Estimation

FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras

Edge-preserving photometric stereo via depth fusion

Depth and Thermal Sensor Fusion to Enhance 3D Thermographic Reconstruction

ToF depth map guided passive stereo based on KNN searching

Temporally Consistent Depth Map Estimation Based On 3d-Mrf

Real-time high resolution 2D-plus-depth video generation using time-of-flight sensor

DELTAR: Depth Estimation from a Light-Weight ToF Sensor and RGB Image

FusionDepth: Complement Self-Supervised Monocular Depth Estimation with Cost Volume

Towards Scale-Aware Self-Supervised Multi-Frame Depth Estimation with IMU Motion Dynamics.

3D Reconstruction Using 3D Registration-Based ToF-Stereo Fusion

Research on TOF camera-based guided binocular fusion technology