Abstract:Depth completion aims to predict dense depth maps with sparse depth measurements from a depth sensor. Currently, Convolutional Neural Network (CNN) based models are the most popular methods applied to depth completion tasks. However, despite the excellent high-end performance, they suffer from a limited representation area. To overcome the drawbacks of CNNs, a more effective and powerful method has been presented: the Transformer, which is an adaptive self-attention setting sequence-to-sequence model. While the standard Transformer quadratically increases the computational cost from the key-query dot-product of input resolution which improperly employs depth completion tasks. In this work, we propose a different window-based Transformer architecture for depth completion tasks named Sparse-to-Dense Transformer (SDformer). The network consists of an input module for the depth map and RGB image features extraction and concatenation, a U-shaped encoder-decoder Transformer for extracting deep features, and a refinement module. Specifically, we first concatenate the depth map features with the RGB image features through the input model. Then, instead of calculating self-attention with the whole feature maps, we apply different window sizes to extract the long-range depth dependencies. Finally, we refine the predicted features from the input module and the U-shaped encoder-decoder Transformer module to get the enriching depth features and employ a convolution layer to obtain the dense depth map. In practice, the SDformer obtains state-of-the-art results against the CNN-based depth completion models with lower computing loads and parameters on the NYU Depth V2 and KITTI DC datasets.

SparseDC: Depth Completion from sparse and non-uniform inputs

Aggregating Feature Point Cloud for Depth Completion

Least Square Estimation Network for Depth Completion

MFF-Net: Towards Efficient Monocular Depth Completion With Multi-Modal Feature Fusion

OMNI-DC: Highly Robust Depth Completion with Multiresolution Depth Integration

Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints

SDC: Spatial Depth Completion for Outdoor Scenes

Sparse Depth Densification for Monocular Depth Estimation

A Sparsity-Invariant Model Via Unifying Depth Prediction and Completion

GeoDC: Geometry-Constrained Depth Completion With Depth Distribution Modeling

DCDepth: Progressive Monocular Depth Estimation in Discrete Cosine Domain

Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera

A Real-Time Semi-Dense Depth-Guided Depth Completion Network

Depth Coefficients for Depth Completion

Deep Sparse Depth Completion Using Joint Depth and Normal Estimation.

Improving Depth Completion Via Depth Feature Upsampling

SDformer: Efficient End-to-End Transformer for Depth Completion

DesNet: Decomposed Scale-Consistent Network for Unsupervised Depth Completion

Towards Domain-agnostic Depth Completion

Dense Depth Completion Based on Piecewise Planar Model

Efficient Depth Completion Network Based on Dynamic Gated Fusion