Abstract:Color guided depth super-resolution (SR) aims to reconstruct a high-resolution (HR) depth image from a low-resolution (LR) one guided by its paired HR color image. However, when the sampling factor is large, color guided depth SR suffers from reconstructing accurate depth edges due to the severe loss of high frequency (HF) components. In this paper, we propose a latent edge guided depth SR network using attention-based hierarchical multi-modal fusion, named LEDSRNet.We extract the hierarchical multi-modal features from HR color and LR depth images, and perform selective fusion to estimate the residual map for depth SR. Firstly, we perform gradient map estimation to generate accurate depth edges from the input HR color image and the interpolated LR depth image, and filter out unnecessary edges in the HR color image while preventing texture copying artifacts in depth SR. Then, we perform depth upsampling to get depth edges from the input LR depth image and refine them guided by gradient features in the latent space. Moreover, we fuse the features extracted from gradient map estimation and depth upsampling to obtain the residual map for depth SR. Finally, we reconstruct SR depth image by adding the residual map to the interpolated LR depth image. We design an attention based multi-level residual block (AMRB) as the basic block for LEDSRNet to extract both shallow and deep features in color and depth images for hierarchical multi-modal fusion. In the loss function, we use a binarized gradient map from the ground truth depth image, i.e. mask map, to calculate the loss for edge and smooth areas separately, preventing excessive smoothing of edge regions in the reconstructed SR depth image. Extensive experiments show that LEDSRNet reconstructs accurate depth edges even in the large sampling factor and achieves the best performance in RMSE with low running time and small model parameters. They indicate that LEDSRNet outperforms state-of-the-art methods in terms of both visual quality and quantitative measurements.

Fast Hierarchical Depth Super-Resolution Via Guided Attention.

Depth Image Super-resolution via Two-Branch Network

Guided Depth Map Super-Resolution Using Recumbent Y Network

Fast, High-Quality Hierarchical Depth-Map Super-Resolution.

Hierarchical Features Driven Residual Learning for Depth Map Super-Resolution.

High-resolution Depth Maps Imaging Via Attention-based Hierarchical Multi-modal Fusion

Progressive Multi-scale Reconstruction for Guided Depth Map Super-Resolution Via Deep Residual Gate Fusion Network

Learning Hierarchical Color Guidance for Depth Map Super-Resolution

Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution.

Joint Depth Map Super-Resolution Method via Deep Hybrid-Cross Guidance Filter

Depth Super-Resolution Via Fully Edge-Augmented Guidance.

Multi-Source Deep Residual Fusion Network for Depth Image Super-resolution

Coarse-to-Fine Depth Super-Resolution with Adaptive RGB-D Feature Attention

Discrete Cosine Transform Network for Guided Depth Map Super-Resolution

Latent Edge Guided Depth Super-Resolution Using Attention-Based Hierarchical Multi-Modal Fusion

Digging into Depth-Adaptive Structure for Guided Depth Super-Resolution

Depth Map Super-Resolution Via Multilevel Recursive Guidance and Progressive Supervision

MSF-Net: Multi-Scale Feedback Reconstruction for Guided Depth Map Super-Resolution

RGB Guided Depth Map Super-Resolution with Coupled U-Net

Super-Resolution for Monocular Depth Estimation with Multi-Scale Sub-Pixel Convolutions and a Smoothness Constraint.

Recurrent Structure Attention Guidance for Depth Super-Resolution