Abstract:Depth cameras have found applications in diverse fields, such as computer vision, artificial intelligence, and video gaming. However, the high latency and low frame rate of existing commodity depth cameras impose limitations on their applications. We propose a fast and accurate depth map reconstruction technique to reduce latency and increase the frame rate in depth cameras. Our approach uses only a commodity depth camera and color camera in a hybrid camera setup; our prototype is implemented using a Kinect Azure depth camera at 30 fps and a high-speed RGB iPhone 11 Pro camera captured at 240 fps. The proposed network, AutoDepthNet, is an encoder-decoder model that captures frames from the high-speed RGB camera and combines them with previous depth frames to reconstruct a stream of high frame rate depth maps. On GPU, with a 480 x 270 output resolution, our system achieves an inference time of 8 ms, enabling real-time use at up to 200 fps with parallel processing. AutoDepthNet can estimate depth values with an average RMS error of 0.076, a 44.5% improvement compared to an optical flow-based comparison method. Our method can also improve depth map quality by estimating depth values for missing and invalidated pixels. The proposed method can be easily applied to existing depth cameras and facilitates the use of depth cameras in applications that require high-speed depth estimation. We also showcase the effectiveness of the framework in upsampling different sparse datasets e.g. video object segmentation. As a demonstration of our method, we integrated our framework into existing body tracking systems and demonstrated the robustness of the proposed method in such applications.

Depth Map Inpainting Using a Fully Convolutional Network

Guided depth enhancement via a fast marching method

Least Square Estimation Network for Depth Completion

Guided inpainting and filtering for Kinect depth maps

Depth map enhancement based on color and depth consistency

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Binocular Depth Estimation Using Convolutional Neural Network With Siamese Branches.

Depth Image Super-resolution via Two-Branch Network

Depth Generation Network: Estimating Real World Depth From Stereo And Depth Images

Real-time Depth Extraction and Multi-View Rendering Algorithm Based on Kinect

High Accuracy Hole Filling for Kinect Depth Maps

Depth incorporating with color improves salient object detection

Mask-adaptive Gated Convolution and Bi-directional Progressive Fusion Network for Depth Completion

Guided Depth Map Super-Resolution Using Recumbent Y Network

Depth Completion Using a View-constrained Deep Prior

Depth-Independent Depth Completion via Least Square Estimation

Depth Estimation from Monocular Images Using Dilated Convolution and Uncertainty Learning.

AGG-Net: Attention Guided Gated-convolutional Network for Depth Image Completion

AutoDepthNet: High Frame Rate Depth Map Reconstruction using Commodity Depth and RGB Cameras

Ddrnet: Depth Map Denoising And Refinement For Consumer Depth Cameras Using Cascaded Cnns

Joint self-supervised and reference-guided learning for depth inpainting