Abstract:The depth images acquired by consumer depth sensors (e.g., Kinect and ToF) usually are of low resolution and insufficient quality. One natural solution is to incorporate a high resolution RGB camera and exploit the statistical correlation of its data and depth. In recent years, both optimization-based and learning-based approaches have been proposed to deal with the guided depth reconstruction problems. In this paper, we introduce a weighted analysis sparse representation (WASR) model for guided depth image enhancement, which can be considered a generalized formulation of a wide range of previous optimization-based models. We unfold the optimization by the WASR model and conduct guided depth reconstruction with dynamically changed stage-wise operations. Such a guidance strategy enables us to dynamically adjust the stage-wise operations that update the depth image, thus improving the reconstruction quality and speed. To learn the stage-wise operations in a task-driven manner, we propose two parameterizations and their corresponding methods: dynamic guidance with Gaussian RBF nonlinearity parameterization (DG-RBF) and dynamic guidance with CNN nonlinearity parameterization (DG-CNN). The network structures of the proposed DG-RBF and DG-CNN methods are designed with the the objective function of our WASR model in mind and the optimal network parameters are learned from paired training data. Such optimization-inspired network architectures enable our models to leverage the previous expertise as well as take benefit from training data. The effectiveness is validated for guided depth image super-resolution and for realistic depth image reconstruction tasks using standard benchmarks. Our DG-RBF and DG-CNN methods achieve the best quantitative results (RMSE) and better visual quality than the state-of-the-art approaches at the time of writing. The code is available at https://github.com/ShuhangGu/GuidedDepthSR.

A Bidirectional Guided Filter Used for RGB-D Maps

Guided inpainting and filtering for Kinect depth maps

Guided depth enhancement via a fast marching method

A Comparative Study on Guided Depth Enhancement

A New Approach of Point Cloud Processing and Scene Segmentation for Guiding the Visually Impaired

Deep Attentional Guided Image Filtering

Decomposed Guided Dynamic Filters for Efficient RGB-Guided Depth Completion

Accurate Depth Sensing by Filtering IR Cost on the Guide of RGB Information

Stereo matching algorithm using improved guided filtering

Full-Image Guided Filtering for Fast Stereo Matching

Fast Color-guided Depth Denoising for RGB-D Images by Graph Filtering

Self‐guided filter for image denoising

A novel guided image filter using orthogonal geodesic distance weight

Hierarchical Joint Bilateral Filtering for Depth Post-Processing

Fast image dehazing using guided filter

Robust Guided Image Filtering

Edge-preserving Guided Filtering Based Cost Aggregation for Stereo Matching.

A Curved Path Extraction Method Using RGB-D Multimodal Data for Single-Edge Guided Navigation in Irregularly Shaped Fields

Improving RealSense by Fusing Color Stereo Vision and Infrared Stereo Vision for the Visually Impaired

Guided Depth Enhancement via Anisotropic Diffusion.

Learned Dynamic Guidance for Depth Image Reconstruction