Embedded event based object detection with spiking neural network

Jonathan Courtois,Pierre-Emmanuel Novac,Edgar Lemaire,Alain Pegatoquet,Benoit Miramond

2024-06-25

Abstract:The complexity of event-based object detection (OD) poses considerable challenges. Spiking Neural Networks (SNNs) show promising results and pave the way for efficient event-based OD. Despite this success, the path to efficient SNNs on embedded devices remains a challenge. This is due to the size of the networks required to accomplish the task and the ability of devices to take advantage of SNNs benefits. Even when "edge" devices are considered, they typically use embedded GPUs that consume tens of watts. In response to these challenges, our research introduces an embedded neuromorphic testbench that utilizes the SPiking Low-power Event-based ArchiTecture (SPLEAT) accelerator. Using an extended version of the Qualia framework, we can train, evaluate, quantize, and deploy spiking neural networks on an FPGA implementation of SPLEAT. We used this testbench to load a state-of-the-art SNN solution, estimate the performance loss associated with deploying the network on dedicated hardware, and run real-world event-based OD on neuromorphic hardware specifically designed for low-power spiking neural networks. Remarkably, our embedded spiking solution, which includes a model with 1.08 million parameters, operates efficiently with 490 mJ per prediction.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

This paper discusses the challenges of implementing event-driven object detection (OD) on embedded devices, particularly using Spiking Neural Networks (SNNs). Despite the energy efficiency and potential of SNNs in handling event data, deploying them on low-power hardware remains difficult due to the requirement of large networks and effective utilization of SNN advantages.

In this study, the authors introduce a low-power event-driven architecture accelerator called SPLEAT and develop an embedded test platform based on the Qualia framework for training, quantizing, and deploying SNNs on FPGAs.

They implement SPLEAT on FPGA using an extended version of Qualia to run an SNN solution with approximately 1.08 million parameters for real-time event-driven object detection in the automotive field. The experiments demonstrate that this embedded spike-based solution consumes only 490 millijoules of energy per prediction, showcasing its efficiency on low-power chips.

The paper also discusses the application of event cameras, SNNs, and OD in autonomous driving, as well as related technologies such as high temporal resolution, wide dynamic range event data, non-volatile neuron models, and event frames. Despite the complexity of SNN training, the difficulty has been reduced through methods like Truncated Backpropagation Through Time (TBPTT) and Surrogate Gradient.

Furthermore, the paper highlights the importance of evaluating SNNs on dedicated hardware as their performance on GPUs is often inferior to hardware specifically designed for SNNs. In conclusion, the authors demonstrate the performance of their embedded SNN in event-driven OD tasks and showcase its potential on low-power embedded hardware, providing new avenues for energy-efficient solutions in future resource-constrained environments.