SD2Event: Self-supervised Learning of Dynamic Detectors and Contextual Descriptors for Event Cameras

Yuan Gao,Yuqing Zhu,Xinjun Li,Yimin Du,Tianzhu Zhang

DOI: https://doi.org/10.1109/cvpr52733.2024.00295

2024-01-01

Abstract:Event cameras offer many advantages over traditional frame-based cameras, such as high dynamic range and low latency. Therefore, event cameras are widely applied in diverse computer vision applications, where event-based key-point detection is a fundamental task. However, achieving robust event-based keypoint detection remains challenging because the ground truth of event keypoints is difficult to obtain, descriptors extracted by CNN usually lack discrimi-native ability in the presence of intense noise, and fixed key-point detectors are limited in detecting varied keypoint patterns. To address these challenges, a novel event-based key-point detection method is proposed by learning dynamic de-tectors and contextual descriptors in a self-supervised manner (SD2Event), including a contextual feature descriptor learning (CFDL) module and a dynamic keypoint detector learning (DKDL) module. The proposed SD2Event enjoys several merits. First, the proposed CFDL module can model long-range contexts efficiently and effectively. Second, the DKDL module generates dynamic keypoint detectors, which can detect keypoints with diverse patterns across various event streams. Third, the proposed self-supervised signals can guide the model's adaptation to event data. Extensive experimental results on three challenging benchmarks show that our proposed method significantly outperforms state-of-the-art event-based keypoint detection methods.