Abstract:Event-based vision sensors encode local pixel-wise brightness changes in streams of events rather than image frames and yield sparse, energy-efficient encodings of scenes, in addition to low latency, high dynamic range, and lack of motion blur. Recent progress in object recognition from event-based sensors has come from conversions of deep neural networks, trained with backpropagation. However, using these approaches for event streams requires a transformation to a synchronous paradigm, which not only loses computational efficiency, but also misses opportunities to extract spatio-temporal features. In this article we propose a hybrid architecture for end-to-end training of deep neural networks for event-based pattern recognition and object detection, combining a spiking neural network (SNN) backbone for efficient event-based feature extraction, and a subsequent analog neural network (ANN) head to solve synchronous classification and detection tasks. This is achieved by combining standard backpropagation with surrogate gradient training to propagate gradients through the SNN. Hybrid SNN-ANNs can be trained without conversion, and result in highly accurate networks that are substantially more computationally efficient than their ANN counterparts. We demonstrate results on event-based classification and object detection datasets, in which only the architecture of the ANN heads need to be adapted to the tasks, and no conversion of the event-based input is necessary. Since ANNs and SNNs require different hardware paradigms to maximize their efficiency, we envision that SNN backbone and ANN head can be executed on different processing units, and thus analyze the necessary bandwidth to communicate between the two parts. Hybrid networks are promising architectures to further advance machine learning approaches for event-based vision, without having to compromise on efficiency.

ASIE: An Asynchronous SNN Inference Engine for AER Events Processing

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

EventF2S: Asynchronous and Sparse Spiking AER Framework using Neuromorphic-Friendly Algorithm

Effective AER Object Classification Using Segmented Probability-Maximization Learning in Spiking Neural Networks

Event-driven Spiking Neural Networks with Spike-Based Learning

An Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

A 0.99-to-4.38 Uj/class Event-Driven Hybrid Neural Network Processor for Full-Spectrum Neural Signal Analyses

A A 22nm 0.43pJ/SOP Sparsity-Aware In-Memory Neuromorphic Computing System with Hybrid Spiking and Artificial Neural Network and Configurable Topology

An Event-Driven Computational System With Spiking Neurons For Object Recognition

Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network

A Bio-Inspired Hierarchical Spiking Neural Network With Reward-Modulated STDP Learning Rule for AER Object Recognition

An Event-Driven Object Recognition Model Using Activated Connected Domain Detection

30.2 A 22nm 0.26nW/Synapse Spike-Driven Spiking Neural Network Processing Unit Using Time-Step-First Dataflow and Sparsity-Adaptive In-Memory Computing

An Asynchronous Multi-core Accelerator for SNN inference

DYNAP-SE2: a scalable multi-core dynamic neuromorphic asynchronous spiking neural network processor

RN-Net: Reservoir Nodes-Enabled Neuromorphic Vision Sensing Network

An 82-Nw 0.53-Pj/sop Clock-Free Spiking Neural Network with 40-Μs Latency for AIoT Wake-Up Functions Using a Multilevel-Event-Driven Bionic Architecture and Computing-in-Memory Technique

Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision

A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perception