Abstract:Remarkable progress has been achieved by current depth completion approaches, which produce dense depth maps from sparse depth maps and corresponding color images. However, the performances of these approaches are limited due to the insufficient feature extractions and fusions. In this work, we propose an efficient multi-modal feature fusion based depth completion framework (MFF-Net), which can efficiently extract and fuse features with different modals in both encoding and decoding processes, thus more depth details with better performance can be obtained. In specific, the encoding process contains three branches where different modals of features from both color and sparse depth input can be extracted, and a multi-feature channel shuffle is utilized to enhance these features thus features with better representation abilities can be obtained. Meanwhile, the decoding process contains two branches to sufficiently fuse the extracted multi-modal features, and a multi-level weighted combination is employed to further enhance and fuse features with different modals, thus leading to more accurate and better refined depth maps. Extensive experiments on different benchmarks demonstrate that we achieve state-of-the-art among online methods. Meanwhile, we further evaluate the predicted dense depth by RGB-D SLAM, which is a commonly used downstream robotic perception task, and higher accuracy on vehicle's trajectory can be obtained in KITTI odometry dataset, which demonstrates the high quality of our depth prediction and the potential of improving the related downstream tasks with depth completion results.

Multiscale Adaptation Fusion Networks for Depth Completion

MFF-Net: Towards Efficient Monocular Depth Completion With Multi-Modal Feature Fusion

Attention-based Multi-modal Fusion Network for Semantic Scene Completion.

An Adaptive Fusion Algorithm for Depth Completion

Gated Recurrent Fusion UNet for Depth Completion

Mask-adaptive Gated Convolution and Bi-directional Progressive Fusion Network for Depth Completion

Deep Sparse Depth Completion Using Multi-Affinity Matrix

Adaptive Context-Aware Multi-Modal Network for Depth Completion

FastCompletion: A Cascade Network with Multiscale Group-Fused Inputs for Real-Time Depth Completion.

Learning an Efficient Multimodal Depth Completion Model

UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion

Gated Cross-Attention Network for Depth Completion

A Multi-Cue Guidance Network for Depth Completion

AF2R Net: Adaptive Feature Fusion and Robust Network for Efficient and Precise Depth Completion

CASIN: Cascading Interaction Network for Robust Depth Sensing with an Auxiliary Task.

A Transformer-Based Image-Guided Depth-Completion Model with Dual-Attention Fusion Module

HMS-Net: Hierarchical Multi-scale Sparsity-invariant Network for Sparse Depth Completion

FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion

MDANet: Multi-Modal Deep Aggregation Network for Depth Completion

RGB-Depth Fusion GAN for Indoor Depth Completion

Multi‐scale Features Fusion from Sparse LiDAR Data and Single Image for Depth Completion