Abstract:There is growing demand for accuracy in image processing and visualization, and the super-resolution (SR) technique for multi-observed RGB-D images has become popular, because it provides space-redundant information and produces a detailed reconstruction even with a large magnification factor. This technique has been thoroughly investigated in recent years. Nevertheless, technical challenges remain, such as finding sub-pixel correspondences with low-resolution (LR) observations, exploiting space-redundant information, formulating space homogeneity constraints, and leveraging cross-image similarities in structures. To address these challenges, this paper proposes a unified optimization framework to estimate both the super-resolved RGB image and the super-resolved depth image from the multi-observed LR RGB-D images using their correlations. Using depth-assisted cross-image correspondences, the RGB image SR problem is formulated as an effective regularization function by incorporating the normalized bilateral total variation regularizer, and it is efficiently solved by a first-order primal-dual algorithm. The depth image SR estimate can be obtained by minimizing a nonlocal regression-based energy, which integrates the structural cues of the super-resolved RGB image in a detail-preserving fashion. Essentially, our unified optimization framework uses the RGB image and depth image as a priori knowledge that the SR process uses for better accuracy. Our extensive experiments on public RGB-D benchmarks and real data and our quantitative comparison with several state-of-the-art methods demonstrate the superiority of our method in terms of accuracy, versatility, and reliability of details and sharp feature preservation.

Learning Complementary Correlations for Depth Super-Resolution With Incomplete Data in Real World

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Depth Image Super-resolution via Two-Branch Network

Depth Map Super-Resolution Based on Joint Dictionary Learning

Channel Attention based Iterative Residual Learning for Depth Map Super-Resolution

DELTAR: Depth Estimation from a Light-Weight ToF Sensor and RGB Image

Joint Depth Map Super-Resolution Method via Deep Hybrid-Cross Guidance Filter

Depth Completion Towards Different Sensor Configurations Via Relative Depth Map Estimation and Scale Recovery

High-Quality 3D Reconstruction with Depth Super-Resolution and Completion

Learning Scene Structure Guidance Via Cross-Task Knowledge Transfer for Single Depth Super-Resolution

Depth image super-resolution using correlation-controlled color guidance and multi-scale symmetric network

Super-Resolution of Multi-Observed RGB-D Images Based on Nonlocal Regression and Total Variation.

Deep Depth Completion from Extremely Sparse Data: A Survey

Deep Depth Super-Resolution : Learning Depth Super-Resolution using Deep Convolutional Neural Network

Hierarchical Features Driven Residual Learning for Depth Map Super-Resolution.

A Joint Learning-Based Method for Multi-view Depth Map Super Resolution

Dense Contrastive Learning and Depth Dynamic Aggregation for Reference-based Super-Resolution.

Recent Advances in Conventional and Deep Learning-Based Depth Completion: A Survey

Towards Fast and Accurate Real-World Depth Super-Resolution: Benchmark Dataset and Baseline

Learning Exhaustive Correlation for Spectral Super-Resolution: Where Spatial-Spectral Attention Meets Linear Dependence

Learning Continuous Depth Representation Via Geometric Spatial Aggregator