Abstract:The depth images acquired by consumer depth sensors (e.g., Kinect and ToF) usually are of low resolution and insufficient quality. One natural solution is to incorporate a high resolution RGB camera and exploit the statistical correlation of its data and depth. In recent years, both optimization-based and learning-based approaches have been proposed to deal with the guided depth reconstruction problems. In this paper, we introduce a weighted analysis sparse representation (WASR) model for guided depth image enhancement, which can be considered a generalized formulation of a wide range of previous optimization-based models. We unfold the optimization by the WASR model and conduct guided depth reconstruction with dynamically changed stage-wise operations. Such a guidance strategy enables us to dynamically adjust the stage-wise operations that update the depth image, thus improving the reconstruction quality and speed. To learn the stage-wise operations in a task-driven manner, we propose two parameterizations and their corresponding methods: dynamic guidance with Gaussian RBF nonlinearity parameterization (DG-RBF) and dynamic guidance with CNN nonlinearity parameterization (DG-CNN). The network structures of the proposed DG-RBF and DG-CNN methods are designed with the the objective function of our WASR model in mind and the optimal network parameters are learned from paired training data. Such optimization-inspired network architectures enable our models to leverage the previous expertise as well as take benefit from training data. The effectiveness is validated for guided depth image super-resolution and for realistic depth image reconstruction tasks using standard benchmarks. Our DG-RBF and DG-CNN methods achieve the best quantitative results (RMSE) and better visual quality than the state-of-the-art approaches at the time of writing. The code is available at https://github.com/ShuhangGu/GuidedDepthSR.

Learning Piecewise Planar Representation for RGB Guided Depth Super-Resolution

Piecewise Planar Super-Resolution for 3D Scene

Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution

Guided Depth Map Super-resolution: A Survey

Guided Depth Map Super-Resolution Using Recumbent Y Network

SRGS: Super-Resolution 3D Gaussian Splatting

The Devil is in the Details: Boosting Guided Depth Super-Resolution via Rethinking Cross-Modal Alignment and Aggregation

MSF-Net: Multi-Scale Feedback Reconstruction for Guided Depth Map Super-Resolution

High-resolution Depth Maps Imaging Via Attention-based Hierarchical Multi-modal Fusion

Joint Depth Map Super-Resolution Method via Deep Hybrid-Cross Guidance Filter

Channel Attention based Iterative Residual Learning for Depth Map Super-Resolution

Learning Hierarchical Color Guidance for Depth Map Super-Resolution

RGB Guided Depth Map Super-Resolution with Coupled U-Net

Learned Dynamic Guidance for Depth Image Reconstruction

DSR-Diff: Depth Map Super-Resolution with Diffusion Model

Single Remote Sensing Image Super-Resolution Via a Generative Adversarial Network with Stratified Dense Sampling and Chain Training

Hierarchical Edge Refinement Network for Guided Depth Map Super-Resolution

SuperGS: Super-Resolution 3D Gaussian Splatting Enhanced by Variational Residual Features and Uncertainty-Augmented Learning

Self-Supervised Learning for RGB-Guided Depth Enhancement by Exploiting the Dependency Between RGB and Depth

Synthesis-guided depth super resolution

Multi-Scale Progressive Fusion Learning for Depth Map Super-Resolution