Abstract:Monocular depth estimation is a crucial task to measure distance relative to a camera, which is important for applications, such as robot navigation and self-driving. Traditional frame-based methods suffer from performance drops due to the limited dynamic range and motion blur. Therefore, recent works leverage novel event cameras to complement or guide the frame modality via frame-event feature fusion. However, event streams exhibit spatial sparsity, leaving some areas unperceived, especially in regions with marginal light changes. Therefore, direct fusion methods, e.g., RAMNet, often ignore the contribution of the most confident regions of each modality. This leads to structural ambiguity in the modality fusion process, thus degrading the depth estimation performance. In this paper, we propose a novel Spatial Reliability-oriented Fusion Network (SRFNet), that can estimate depth with fine-grained structure at both daytime and nighttime. Our method consists of two key technical components. Firstly, we propose an attention-based interactive fusion (AIF) module that applies spatial priors of events and frames as the initial masks and learns the consensus regions to guide the inter-modal feature fusion. The fused feature are then fed back to enhance the frame and event feature learning. Meanwhile, it utilizes an output head to generate a fused mask, which is iteratively updated for learning consensual spatial priors. Secondly, we propose the Reliability-oriented Depth Refinement (RDR) module to estimate dense depth with the fine-grained structure based on the fused features and masks. We evaluate the effectiveness of our method on the synthetic and real-world datasets, which shows that, even without pretraining, our method outperforms the prior methods, e.g., RAMNet, especially in night scenes. Our project homepage: <a class="link-external link-https" href="https://vlislab22.github.io/SRFNet" rel="external noopener nofollow">this https URL</a>.

Data Association Between Event Streams and Intensity Frames Under Diverse Baselines.

Intensity/Inertial Integration-Aided Feature Tracking on Event Cameras

Background-aware Siamese Network Tracking Based on Salient Feature Fusion

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

MFF-Net: Towards Efficient Monocular Depth Completion With Multi-Modal Feature Fusion

Event Stream Learning Using Spatio-Temporal Event Surface

Local Optimized and Scalable Frame-to-model SLAM

EMatch: A Unified Framework for Event-based Optical Flow and Stereo Matching

EvIntSR-Net: Event Guided Multiple Latent Frames Reconstruction and Super-resolution

SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

Learning to Estimate Two Dense Depths from LiDAR and Event Data

Deep Representation Via Convolutional Neural Network for Classification of Spatiotemporal Event Streams

SRFNet: Monocular Depth Estimation with Fine-grained Structure via Spatial Reliability-oriented Fusion of Frames and Events

TSFE-Net: Two-Stream Feature Extraction Networks for Active Stereo Matching

Deeply-fused Attentive Network for Stereo Matching

Dense Depth from Event Focal Stack

Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks

Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking

Long-term Frame-Event Visual Tracking: Benchmark Dataset and Baseline

Exploring the Mutual Influence between Self-Supervised Single-Frame and Multi-Frame Depth Estimation

LongShortNet: Exploring Temporal and Semantic Features Fusion in Streaming Perception