Abstract:With the success of deep learning, object recognition systems that can be deployed for real-world applications are becoming commonplace. However, inference that needs to largely take place on the `edge' (not processed on servers), is a highly computational and memory intensive workload, making it intractable for low-power mobile nodes and remote security applications. To address this challenge, this paper proposes a low-power (5W) end-to-end neuromorphic framework for object tracking and classification using event-based cameras that possess desirable properties such as low power consumption (5-14 mW) and high dynamic range (120 dB). Nonetheless, unlike traditional approaches of using event-by-event processing, this work uses a mixed frame and event approach to get energy savings with high performance. Using a frame-based region proposal method based on the density of foreground events, a hardware-friendly object tracking is implemented using the apparent object velocity while tackling occlusion scenarios. For low-power classification of the tracked objects, the event camera is interfaced to IBM TrueNorth, which is time-multiplexed to tackle up to eight instances for a traffic monitoring application. The frame-based object track input is converted back to spikes for Truenorth classification via the energy efficient deep network (EEDN) pipeline. Using originally collected datasets, we train the TrueNorth model on the hardware track outputs, instead of using ground truth object locations as commonly done, and demonstrate the efficacy of our system to handle practical surveillance scenarios. Finally, we compare the proposed methodologies to state-of-the-art event-based systems for object tracking and classification, and demonstrate the use case of our neuromorphic approach for low-power applications without sacrificing on performance.

Multi-resolution Rescored ByteTrack for Video Object Detection on Ultra-low-power Embedded Systems

High-performance Reconfigurable DNN Accelerator on a Bandwidth-limited Embedded System

P2M-DeTrack: Processing-in-Pixel-in-Memory for Energy-efficient and Real-Time Multi-Object Detection and Tracking

TOD: Transprecise Object Detection to Maximise Real-Time Accuracy on the Edge

A low-power end-to-end hybrid neuromorphic framework for surveillance applications

Adaptive Confidence Threshold for ByteTrack in Multi-Object Tracking

Deep Learning-Based Multiple Object Visual Tracking on Embedded System for IoT and Mobile Edge Computing Applications

EdgeNet: Balancing Accuracy and Performance for Edge-based Convolutional Neural Network Object Detectors

An Adaptive Video Acquisition Scheme for Object Tracking and its Performance Optimization

ByteTrackV2: 2D and 3D Multi-Object Tracking by Associating Every Detection Box

DSORT-MCU: Detecting Small Objects in Real-Time on Microcontroller Units

DeepScale: Online Frame Size Adaptation for Multi-object Tracking on Smart Cameras and Edge Servers

Enhanced TLD-based video object-tracking implementation tested on embedded platforms

SkyNet: a Hardware-Efficient Method for Object Detection and Tracking on Embedded Systems

Towards Real-Time Object Detection on Embedded Systems.

Finding a Needle in a Haystack: Tiny Flying Object Detection in 4K Videos using a Joint Detection-and-Tracking Approach

MobileNet-JDE: a lightweight multi-object tracking model for embedded systems

DEFT: Detection Embeddings for Tracking

High-Speed tracking-by-detection without using image information

HopTrack: A Real-time Multi-Object Tracking System for Embedded Devices

Real-Time Embedded Implementation of Improved Object Detector for Resource-Constrained Devices