Abstract:The depth images acquired by consumer depth sensors (e.g., Kinect and ToF) usually are of low resolution and insufficient quality. One natural solution is to incorporate a high resolution RGB camera and exploit the statistical correlation of its data and depth. In recent years, both optimization-based and learning-based approaches have been proposed to deal with the guided depth reconstruction problems. In this paper, we introduce a weighted analysis sparse representation (WASR) model for guided depth image enhancement, which can be considered a generalized formulation of a wide range of previous optimization-based models. We unfold the optimization by the WASR model and conduct guided depth reconstruction with dynamically changed stage-wise operations. Such a guidance strategy enables us to dynamically adjust the stage-wise operations that update the depth image, thus improving the reconstruction quality and speed. To learn the stage-wise operations in a task-driven manner, we propose two parameterizations and their corresponding methods: dynamic guidance with Gaussian RBF nonlinearity parameterization (DG-RBF) and dynamic guidance with CNN nonlinearity parameterization (DG-CNN). The network structures of the proposed DG-RBF and DG-CNN methods are designed with the the objective function of our WASR model in mind and the optimal network parameters are learned from paired training data. Such optimization-inspired network architectures enable our models to leverage the previous expertise as well as take benefit from training data. The effectiveness is validated for guided depth image super-resolution and for realistic depth image reconstruction tasks using standard benchmarks. Our DG-RBF and DG-CNN methods achieve the best quantitative results (RMSE) and better visual quality than the state-of-the-art approaches at the time of writing. The code is available at https://github.com/ShuhangGu/GuidedDepthSR.

Small-Data-Driven Multitask Computational Guidance for Constrained Impact

Understanding and Improving Training-free Loss-based Diffusion Guidance

Guided Learning: Lubricating End-to-End Modeling for Multi-stage Decision-making

A Model-Agnostic Approach to Mitigate Gradient Interference for Multi-Task Learning

The Unreasonable Effectiveness of Guidance for Diffusion Models

Dreamguider: Improved Training free Diffusion-based Conditional Generation

Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective

Boosting auxiliary task guidance: a probabilistic approach

Mitigating Gradient Conflicts Via Expert Squads in Multi-Task Learning

Gradient Guidance for Diffusion Models: An Optimization Perspective

Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling

Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation

Studying How to Efficiently and Effectively Guide Models with Explanations

Efficient Controllable Multi-Task Architectures

GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal Conditioned Policy

Gradient Surgery for Multi-Task Learning

A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloading

Learning Diverse Robot Striking Motions with Diffusion Models and Kinematically Constrained Gradient Guidance

Learned Dynamic Guidance for Depth Image Reconstruction

FAMO: Fast Adaptive Multitask Optimization

Towards Large-Scale Evolutionary Multi-Tasking: A GPU-Based Paradigm