Abstract:The rise of mobility, IoT and wearables has shifted processing to the edge of the sensors, driven by the need to reduce latency, communication costs and overall energy consumption. While deep learning models have achieved remarkable results in various domains, their deployment at the edge for real-time applications remains computationally expensive. Neuromorphic computing emerges as a promising paradigm shift, characterized by co-localized memory and computing as well as event-driven asynchronous sensing and processing. In this work, we demonstrate the possibility of solving the ubiquitous computer vision task of object detection at the edge with low-power requirements, using the event-based N-Caltech101 dataset. We present the first instance of an on-chip spiking neural network for event-based face detection deployed on the SynSense Speck neuromorphic chip, which comprises both an event-based sensor and a spike-based asynchronous processor implementing Integrate-and-Fire neurons. We show how to reduce precision discrepancies between off-chip clock-driven simulation used for training and on-chip event-driven inference. This involves using a multi-spike version of the Integrate-and-Fire neuron on simulation, where spikes carry values that are proportional to the extent the membrane potential exceeds the firing threshold. We propose a robust strategy to train spiking neural networks with back-propagation through time using multi-spike activation and firing rate regularization and demonstrate how to decode output spikes into bounding boxes. We show that the power consumption of the chip is directly proportional to the number of synaptic operations in the spiking neural network, and we explore the trade-off between power consumption and detection precision with different firing rate regularization, achieving an on-chip face detection mAP[0.5] of ~0.6 while consuming only ~20 mW.

An Ultra-Low-Power Image Signal Processor for Hierarchical Image Recognition with Deep Neural Networks

A 170μw Image Signal Processor Enabling Hierarchical Image Recognition for Intelligence at the Edge

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

Senputing: An Ultra-Low-Power Always-On Vision Perception Chip Featuring the Deep Fusion of Sensing and Computing

Millimeter-Scale Ultra-Low-Power Imaging System for Intelligent Edge Monitoring

Processing Near Sensor Architecture in Mixed-Signal Domain with CMOS Image Sensor of Convolutional-Kernel-Readout Method

A 0.99-to-4.38 Uj/class Event-Driven Hybrid Neural Network Processor for Full-Spectrum Neural Signal Analyses

Ultra-low Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

A 5.9μw Ultra-Low-Power Dual-Resolution CIS Chip of Sensing-with-Computing for Always-on Intelligent Visual Devices

A low-power end-to-end hybrid neuromorphic framework for surveillance applications

Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

A Tightly Coupled AI-ISP Vision Processor

An Ultralow-Voltage Sensor Node Processor with Diverse Hardware Acceleration and Cognitive Sampling for Intelligent Sensing

A 0.8 V Intelligent Vision Sensor With Tiny Convolutional Neural Network and Programmable Weights Using Mixed-Mode Processing-in-Sensor Technique for Image Classification

A 1.8mW Perception Chip with Near-Sensor Processing Scheme for Low-Power AIoT Applications

A 2.17μw@120fps Ultra-Low-Power Dual-Mode CMOS Image Sensor with Senputing Architecture

An 1.38nj/inference Clock-Free Mixed-Signal Neuromorphic Architecture Using ReL-PSP Function and Computing-in-Memory.

Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipeline

Low-power event-based face detection with asynchronous neuromorphic hardware

A 4.57 Μw@120fps Vision System of Sensing with Computing for BNN-Based Perception Applications

A 68 mw 2.2 Tops/w low bit-width and multiplierless DCNN object detection processor for visually impaired people