Abstract:Spiking camera is a bio-inspired vision sensor that mimics the sampling mechanism of the primate fovea, which has shown great potential for capturing high-speed dynamic scenes with a sampling rate of 40,000Hz. Unlike conventional digital cameras, the spiking camera continuously captures photons and outputs asynchronous binary spikes that encode time, location, and light intensity. Because of the different sampling mechanisms, the off-the-shelf image-based algorithms for digital cameras are unsuitable for spike streams generated by the spiking camera. Therefore, it is of particular interest to develop novel, spike-aware algorithms for common computer vision tasks. In this paper, we focus on the depth estimation task, which is challenging due to the natural properties of spike streams, such as irregularity, continuity, and spatialtemporal correlation, and has not been explored for the spiking camera. We present Spike Transformer (Spike-T), a novel paradigm for learning spike data and estimating monocular depth from continuous spike streams. To fit spike data to Transformer, we present an input spike embedding equipped with a spatio-temporal patch partition module to maintain features from both spatial and temporal domains. Furthermore, we build two spike-based depth datasets. One is synthetic, and the other is captured by a real spiking camera. Experimental results demonstrate that the proposed Spike-T can favorably predict the scene's depth and consistently outperform its direct competitors. More importantly, the representation learned by Spike-T transfers well to the unseen real data, indicating the generalization of Spike-T to real-world scenarios. To our best knowledge, this is the first time that directly depth estimation from spike streams becomes possible. Code and Datasets are available at https://github.com/Leozhangjiyuan/MDE-SpildngCamera.

Spike Coding for Dynamic Vision Sensors

Spike Coding for Dynamic Vision Sensor in Intelligent Driving

Spike Coding: Towards Lossy Compression for Dynamic Vision Sensor.

An Efficient Coding Method for Spike Camera using Inter-Spike Intervals

Hybrid Coding of Spatiotemporal Spike Data for a Bio-Inspired Camera

SpikeCodec: An End-to-end Learned Compression Framework for Spiking Camera

An Event Coding Method Based on Frame Images with Dynamic Vision Sensor Modeling

Spike Camera and Its Coding Methods

A Lossless Compression Algorithm with Hardware Implementation for Dynamic Vision Sensor

Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations

Unifying Spike Perception and Prediction: A Compact Spike Representation Model Using Multi-scale Correlation

SpikeCV: Open a Continuous Computer Vision Era

NeuSpike-Net: High Speed Video Reconstruction via Bio-inspired Neuromorphic Cameras

Robust Transcoding Sensory Information with Neural Spikes

Motion Estimation for Spike Camera Data Sequence Via Spike Interval Analysis.

Spike-Based Vision for Autonomous Driving Scenarios

Enhancing Motion Deblurring in High-Speed Scenes with Spike Streams.

Spike Signal Reconstruction Based on Inter-Spike Similarity

Spike Transformer: Monocular Depth Estimation for Spiking Camera

Learning Temporal-Ordered Representation for Spike Streams Based on Discrete Wavelet Transforms.

Spike Attention Coding for Spiking Neural Networks.