Abstract:Deep learning inference that needs to largely take place on the 'edge' is a highly computational and memory intensive workload, making it intractable for low-power, embedded platforms such as mobile nodes and remote security applications. To address this challenge, this paper proposes a real-time, hybrid neuromorphic framework for object tracking and classification using event-based cameras that possess properties such as low-power consumption (5-14 mW) and high dynamic range (120 dB). Nonetheless, unlike traditional approaches of using event-by-event processing, this work uses a mixed frame and event approach to get energy savings with high performance. Using a frame-based region proposal method based on the density of foreground events, a hardware-friendly object tracking scheme is implemented using the apparent object velocity while tackling occlusion scenarios. The object track input is converted back to spikes for TrueNorth classification via the energy-efficient deep network (EEDN) pipeline. Using originally collected datasets, we train the TrueNorth model on the hardware track outputs, instead of using ground truth object locations as commonly done, and demonstrate the ability of our system to handle practical surveillance scenarios. As an optional paradigm, to exploit the low latency and asynchronous nature of neuromorphic vision sensors (NVS), we also propose a continuous-time tracker with C++ implementation where each event is processed individually. Thereby, we extensively compare the proposed methodologies to state-of-the-art event-based and frame-based methods for object tracking and classification, and demonstrate the use case of our neuromorphic approach for real-time and embedded applications without sacrificing performance. Finally, we also showcase the efficacy of the proposed system to a standard RGB camera setup when evaluated over several hours of traffic recordings.

A compact neuromorphic system for ultra energy-efficient, on-device robot localization

31.1 A 65nm 8.79TOPS/W 23.82mW Mixed-Signal Oscillator-Based NeuroSLAM Accelerator for Applications in Edge Robotics

An Embedded Tracking System With Neural Network Accelerator

A Navigation Cognitive System Driven by Hierarchical Spiking Neural Network.

An All-Neuron Spiking 2-D Path Integration and Map Representation Model Implemented on Neuromorphic Chips

Low-power event-based face detection with asynchronous neuromorphic hardware

A low-power end-to-end hybrid neuromorphic framework for surveillance applications

Ensembles of Compact, Region-specific & Regularized Spiking Neural Networks for Scalable Place Recognition

Neuromorphic force-control in an industrial task: validating energy and latency benefits

An Optical Neuromorphic Sensor with High Uniformity and High Linearity for Indoor Visible Light Localization

Efficient Spatio-Temporal Signal Recognition on Edge Devices Using PointLCA-Net

Fully Asynchronous Neuromorphic Perception for Mobile Robot Dodging with Loihi Chips

Neuromorphic Near-Sensor Computing: From Event-Based Sensing to Edge Learning

A Hybrid Neuromorphic Object Tracking and Classification Framework for Real-time Systems

Brain-inspired multimodal hybrid neural network for robot place recognition

Emulating Brain-like Rapid Learning in Neuromorphic Edge Computing

μBrain: An Event-Driven and Fully Synthesizable Architecture for Spiking Neural Networks

Detection of Fast-Moving Objects with Neuromorphic Hardware

Energy Efficiency of Machine Learning in Embedded Systems Using Neuromorphic Hardware

Obstacle Avoidance and Target Acquisition for Robot Navigation Using a Mixed Signal Analog/Digital Neuromorphic Processing System

Rhythmic Representations: Learning Periodic Patterns for Scalable Place Recognition at a Sub-Linear Storage Cost