Abstract:Spiking camera is a bio-inspired vision sensor that mimics the sampling mechanism of the primate fovea, which has shown great potential for capturing high-speed dynamic scenes with a sampling rate of 40,000Hz. Unlike conventional digital cameras, the spiking camera continuously captures photons and outputs asynchronous binary spikes that encode time, location, and light intensity. Because of the different sampling mechanisms, the off-the-shelf image-based algorithms for digital cameras are unsuitable for spike streams generated by the spiking camera. Therefore, it is of particular interest to develop novel, spike-aware algorithms for common computer vision tasks. In this paper, we focus on the depth estimation task, which is challenging due to the natural properties of spike streams, such as irregularity, continuity, and spatialtemporal correlation, and has not been explored for the spiking camera. We present Spike Transformer (Spike-T), a novel paradigm for learning spike data and estimating monocular depth from continuous spike streams. To fit spike data to Transformer, we present an input spike embedding equipped with a spatio-temporal patch partition module to maintain features from both spatial and temporal domains. Furthermore, we build two spike-based depth datasets. One is synthetic, and the other is captured by a real spiking camera. Experimental results demonstrate that the proposed Spike-T can favorably predict the scene's depth and consistently outperform its direct competitors. More importantly, the representation learned by Spike-T transfers well to the unseen real data, indicating the generalization of Spike-T to real-world scenarios. To our best knowledge, this is the first time that directly depth estimation from spike streams becomes possible. Code and Datasets are available at https://github.com/Leozhangjiyuan/MDE-SpildngCamera.

Unifying Spike Perception and Prediction: A Compact Spike Representation Model Using Multi-scale Correlation

Toward Efficient Processing and Learning with Spikes: New Approaches for Multispike Learning

Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations

Learning Temporal-Ordered Representation for Spike Streams Based on Discrete Wavelet Transforms.

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

High-speed and High-quality Vision Reconstruction of Spike Camera with Spike Stability Theorem

A Spike-Timing Based Integrated Model for Pattern Recognition

Learning Stereo Depth Estimation with Bio-Inspired Spike Cameras

Spike Transformer: Monocular Depth Estimation for Spiking Camera

An Efficient Coding Method for Spike Camera using Inter-Spike Intervals

NeuSpike-Net: High Speed Video Reconstruction via Bio-inspired Neuromorphic Cameras

Robust Environmental Sound Recognition with Sparse Key-point Encoding and Efficient Multi-spike Learning.

Sparse Temporal Encoding of Visual Features for Robust Object Recognition by Spiking Neurons

A Multi-Scale Threshold Spiking Mechanism Based on Spike Camera

Spike Signal Reconstruction Based on Inter-Spike Similarity

SpikeCV: Open a Continuous Computer Vision Era

Spike Camera Image Reconstruction Using Deep Spiking Neural Networks

Ultra-high Temporal Resolution Visual Reconstruction from a Fovea-like Spike Camera via Spiking Neuron Model

Recognizing Ultra-High-Speed Moving Objects with Bio-Inspired Spike Camera

Spike-Visnet: A Novel Framework for Visual Recognition with Focuslayer-Stdp Learning

Super-Resolution Reconstruction from Bayer-Pattern Spike Streams