SteeredMarigold: Steering Diffusion Towards Depth Completion of Largely Incomplete Depth Maps

Jakub Gregorek,Lazaros Nalpantidis

2024-09-16

Abstract:Even if the depth maps captured by RGB-D sensors deployed in real environments are often characterized by large areas missing valid depth measurements, the vast majority of depth completion methods still assumes depth values covering all areas of the scene. To address this limitation, we introduce SteeredMarigold, a training-free, zero-shot depth completion method capable of producing metric dense depth, even for largely incomplete depth maps. SteeredMarigold achieves this by using the available sparse depth points as conditions to steer a denoising diffusion probabilistic model. Our method outperforms relevant top-performing methods on the NYUv2 dataset, in tests where no depth was provided for a large area, achieving state-of-art performance and exhibiting remarkable robustness against depth map incompleteness. Our code will be publicly available.

Robotics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of obtaining high-resolution, dense depth perception for robots in situations where some depth measurements are missing. Specifically, the paper proposes a method called **SteeredMarigold**, which is a training-free, zero-shot depth completion method capable of generating dense metric depth maps even when most of the depth map is missing.

The main issues include:
1. **Non-uniform Sparsity in Depth Completion**: Existing depth completion methods assume a relatively uniform distribution of depth values, but in reality, depth maps captured by RGB-D sensors often have large areas of missing data.
2. **Limitations of Existing Methods**: Most depth completion methods perform poorly when faced with non-uniformly distributed depth data, while monocular depth estimation methods completely ignore existing depth data, which is too risky for robotic applications.

The SteeredMarigold method utilizes existing sparse depth points as conditions to guide the Denoising Diffusion Probabilistic Model (DDPM), thereby achieving effective completion of depth maps with large missing areas. Experimental results show that this method outperforms current state-of-the-art methods on the NYUv2 dataset and performs excellently when dealing with a large amount of missing depth data.