Abstract:The depth images acquired by consumer depth sensors (e.g., Kinect and ToF) usually are of low resolution and insufficient quality. One natural solution is to incorporate a high resolution RGB camera and exploit the statistical correlation of its data and depth. In recent years, both optimization-based and learning-based approaches have been proposed to deal with the guided depth reconstruction problems. In this paper, we introduce a weighted analysis sparse representation (WASR) model for guided depth image enhancement, which can be considered a generalized formulation of a wide range of previous optimization-based models. We unfold the optimization by the WASR model and conduct guided depth reconstruction with dynamically changed stage-wise operations. Such a guidance strategy enables us to dynamically adjust the stage-wise operations that update the depth image, thus improving the reconstruction quality and speed. To learn the stage-wise operations in a task-driven manner, we propose two parameterizations and their corresponding methods: dynamic guidance with Gaussian RBF nonlinearity parameterization (DG-RBF) and dynamic guidance with CNN nonlinearity parameterization (DG-CNN). The network structures of the proposed DG-RBF and DG-CNN methods are designed with the the objective function of our WASR model in mind and the optimal network parameters are learned from paired training data. Such optimization-inspired network architectures enable our models to leverage the previous expertise as well as take benefit from training data. The effectiveness is validated for guided depth image super-resolution and for realistic depth image reconstruction tasks using standard benchmarks. Our DG-RBF and DG-CNN methods achieve the best quantitative results (RMSE) and better visual quality than the state-of-the-art approaches at the time of writing. The code is available at https://github.com/ShuhangGu/GuidedDepthSR.

Hierarchical Edge Refinement Network for Guided Depth Map Super-Resolution

Depth Image Super-resolution via Two-Branch Network

Guided Depth Map Super-Resolution Using Recumbent Y Network

A Comparative Study on Guided Depth Enhancement

Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution

Learning Hierarchical Color Guidance for Depth Map Super-Resolution

The Devil is in the Details: Boosting Guided Depth Super-Resolution via Rethinking Cross-Modal Alignment and Aggregation

Super-resolution Reconstruction of Depth Image Based on Edge-Selected Deep Residual Network.

MSF-Net: Multi-Scale Feedback Reconstruction for Guided Depth Map Super-Resolution

Guided Depth Map Super-resolution: A Survey

Depth Cue Enhancement and Guidance Network for RGB-D Salient Object Detection

Remote Sensing Single-Image Resolution Improvement Using A Deep Gradient-Aware Network with Image-Specific Enhancement

High-resolution Depth Maps Imaging Via Attention-based Hierarchical Multi-modal Fusion

Real-GDSR: Real-World Guided DSM Super-Resolution via Edge-Enhancing Residual Network

Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution.

Learning Piecewise Planar Representation for RGB Guided Depth Super-Resolution

Learned Dynamic Guidance for Depth Image Reconstruction

Joint Depth Map Super-Resolution Method via Deep Hybrid-Cross Guidance Filter

RGB Guided Depth Map Super-Resolution with Coupled U-Net

Channel Attention based Iterative Residual Learning for Depth Map Super-Resolution

Dense Contrastive Learning and Depth Dynamic Aggregation for Reference-based Super-Resolution.