Abstract:Depth cameras have found applications in diverse fields, such as computer vision, artificial intelligence, and video gaming. However, the high latency and low frame rate of existing commodity depth cameras impose limitations on their applications. We propose a fast and accurate depth map reconstruction technique to reduce latency and increase the frame rate in depth cameras. Our approach uses only a commodity depth camera and color camera in a hybrid camera setup; our prototype is implemented using a Kinect Azure depth camera at 30 fps and a high-speed RGB iPhone 11 Pro camera captured at 240 fps. The proposed network, AutoDepthNet, is an encoder-decoder model that captures frames from the high-speed RGB camera and combines them with previous depth frames to reconstruct a stream of high frame rate depth maps. On GPU, with a 480 x 270 output resolution, our system achieves an inference time of 8 ms, enabling real-time use at up to 200 fps with parallel processing. AutoDepthNet can estimate depth values with an average RMS error of 0.076, a 44.5% improvement compared to an optical flow-based comparison method. Our method can also improve depth map quality by estimating depth values for missing and invalidated pixels. The proposed method can be easily applied to existing depth cameras and facilitates the use of depth cameras in applications that require high-speed depth estimation. We also showcase the effectiveness of the framework in upsampling different sparse datasets e.g. video object segmentation. As a demonstration of our method, we integrated our framework into existing body tracking systems and demonstrated the robustness of the proposed method in such applications.

Real-Time Depth Image Acquisition and Restoration for Image Based Rendering and Processing Systems

Superpixel-based Color–depth Restoration and Dynamic Environment Modeling for Kinect-assisted Image-Based Rendering Systems

Guided depth enhancement via a fast marching method

Depth map enhancement based on color and depth consistency

Calibration-free Deep Optics for Depth Estimation with Precise Simulation

Real-time Depth Extraction and Multi-View Rendering Algorithm Based on Kinect

Depth Map Inpainting Using a Fully Convolutional Network

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

Data Driven Robust Image Guided Depth Map Restoration

Depth-color fusion strategy for 3-D scene modeling with Kinect

High-Quality Depth Recovery Via Interactive Multi-view Stereo

Texture-Assisted Kinect Depth Inpainting

Kinect Depth Recovery Using a Color-Guided, Region-Adaptive, and Depth-Selective Framework

Depth-Aided Inpainting for Disocclusion Restoration of Multi-View Images Using Depth-Image-Based Rendering

Improving RealSense by Fusing Color Stereo Vision and Infrared Stereo Vision for the Visually Impaired

DELTAR: Depth Estimation from a Light-Weight ToF Sensor and RGB Image

Recovering Consistent Video Depth Maps Via Bundle Optimization

Depth Recovery for Kinect Sensor Using Contour-Guided Adaptive Morphology Filter

A Unified Model For Improving Depth Accuracy In Kinect Sensor

Robust, Efficient Depth Reconstruction with Hierarchical Confidence-Based Matching.

AutoDepthNet: High Frame Rate Depth Map Reconstruction using Commodity Depth and RGB Cameras