Hybrid Coding of Spatiotemporal Spike Data for a Bio-Inspired Camera

Lin Zhu,Siwei Dong,Tiejun Huang,Yonghong Tian

DOI: https://doi.org/10.1109/tcsvt.2020.3032014

IF: 5.859

2020-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Recently, a novel bio-inspired camera was developed by mimicking the retina fovea to continuously accumulate luminance intensity and then fire spikes once the dispatch threshold is reached. In contrast to the conventional frame-based cameras and the emerging dynamic vision sensors, this spike camera has shown remarkable advantages in capturing fast-moving scenes in a frame-free manner with full texture reconstruction capabilities. However, the ultra-high temporal resolution makes the transmission or storage of the output data of spike camera (referred to as spike data) quite difficult. To address the above challenges, we propose a unified lossy spike coding framework, which exploits the motion patterns hidden in the spike data distribution to design the motion-fidelity coding modes for the first time. We investigate the spatiotemporal distribution of spike data and propose an intensity-based measurement of the spike train distance. Then, the adaptive polyhedron partitioning is proposed to deal with the spike data with different motion characteristics. Finally, the intra-/inter-polyhedron prediction with spike-time and spike-rate modes, transform and multi-layer quantization are proposed and introduced into the codec. We also construct a PKU-Spike dataset captured by the spike camera to evaluate the compression performance. The experimental results on the dataset demonstrate that the proposed approach is effective in compressing such spike data while maintaining the visual fidelity especially for high-speed scenarios.