Abstract:Monocular depth estimation is a crucial task to measure distance relative to a camera, which is important for applications, such as robot navigation and self-driving. Traditional frame-based methods suffer from performance drops due to the limited dynamic range and motion blur. Therefore, recent works leverage novel event cameras to complement or guide the frame modality via frame-event feature fusion. However, event streams exhibit spatial sparsity, leaving some areas unperceived, especially in regions with marginal light changes. Therefore, direct fusion methods, e.g., RAMNet, often ignore the contribution of the most confident regions of each modality. This leads to structural ambiguity in the modality fusion process, thus degrading the depth estimation performance. In this paper, we propose a novel Spatial Reliability-oriented Fusion Network (SRFNet), that can estimate depth with fine-grained structure at both daytime and nighttime. Our method consists of two key technical components. Firstly, we propose an attention-based interactive fusion (AIF) module that applies spatial priors of events and frames as the initial masks and learns the consensus regions to guide the inter-modal feature fusion. The fused feature are then fed back to enhance the frame and event feature learning. Meanwhile, it utilizes an output head to generate a fused mask, which is iteratively updated for learning consensual spatial priors. Secondly, we propose the Reliability-oriented Depth Refinement (RDR) module to estimate dense depth with the fine-grained structure based on the fused features and masks. We evaluate the effectiveness of our method on the synthetic and real-world datasets, which shows that, even without pretraining, our method outperforms the prior methods, e.g., RAMNet, especially in night scenes. Our project homepage: <a class="link-external link-https" href="https://vlislab22.github.io/SRFNet" rel="external noopener nofollow">this https URL</a>.

CRF-based depth refinement with hybrid depth information

CRF-Based Reconstruction from Narrow-Baseline Image Sequences.

A Comparative Study on Guided Depth Enhancement

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Depth Upsampling Method Via Markov Random Fields Without Edge-Misaligned Artifacts

Multi-Source Deep Residual Fusion Network for Depth Image Super-resolution

Guided Depth Map Super-Resolution Using Recumbent Y Network

Depth Map Super-Resolution Based on Joint Dictionary Learning

Depth and Surface Normal Estimation from Monocular Images Using Regression on Deep Features and Hierarchical CRFs.

Detail-Enhancing Framework for Reference-Based Image Super-Resolution

Single Image Depth Estimation with Normal Guided Scale Invariant Deep Convolutional Fields

Depth Super-Resolution Via Joint Color-Guided Internal and External Regularizations

Joint Depth Map Super-Resolution Method via Deep Hybrid-Cross Guidance Filter

Enhanced Feature Refinement Network Based on Depthwise Separable Convolution for Lightweight Image Super-Resolution

Temporally Consistent Depth Map Prediction Using Deep Convolutional Neural Network and Spatial-Temporal Conditional Random Field

Super-resolution Reconstruction of Depth Image Based on Edge-Selected Deep Residual Network.

Multiscale and Multidirection Depth Map Super Resolution with Semantic Inference

Depth Super-Resolution Using Joint Adaptive Weighted Least Squares and Patching Gradient.

FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion

CRF360D: Monocular 360 Depth Estimation via Spherical Fully-Connected CRFs

SRFNet: Monocular Depth Estimation with Fine-grained Structure via Spatial Reliability-oriented Fusion of Frames and Events