TRIP: Trainable Region-of-Interest Prediction for Hardware-Efficient Neuromorphic Processing on Event-based Vision

Cina Arjmand,Yingfu Xu,Kevin Shidqi,Alexandra F. Dobrita,Kanishkan Vadivel,Paul Detterer,Manolis Sifalakis,Amirreza Yousefzadeh,Guangzhi Tang

2024-06-25

Abstract:Neuromorphic processors are well-suited for efficiently handling sparse events from event-based cameras. However, they face significant challenges in the growth of computing demand and hardware costs as the input resolution increases. This paper proposes the Trainable Region-of-Interest Prediction (TRIP), the first hardware-efficient hard attention framework for event-based vision processing on a neuromorphic processor. Our TRIP framework actively produces low-resolution Region-of-Interest (ROIs) for efficient and accurate classification. The framework exploits sparse events' inherent low information density to reduce the overhead of ROI prediction. We introduced extensive hardware-aware optimizations for TRIP and implemented the hardware-optimized algorithm on the SENECA neuromorphic processor. We utilized multiple event-based classification datasets for evaluation. Our approach achieves state-of-the-art accuracies in all datasets and produces reasonable ROIs with varying locations and sizes. On the DvsGesture dataset, our solution requires 46x less computation than the state-of-the-art while achieving higher accuracy. Furthermore, TRIP enables more than 2x latency and energy improvements on the SENECA neuromorphic processor compared to the conventional solution.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

This paper proposes a solution to the problem of increasing computational demands and hardware costs in high-resolution event-based visual processing. Traditional neuromorphic processors are efficient in processing sparse data from event-based cameras, but they face challenges in terms of computation and hardware costs with the increase in input resolution. To address this, the paper introduces the Trainable Region-of-Interest Prediction (TRIP) framework, which is the first hardware-efficient and attention-based framework for neuromorphic processors in event-based visual processing. TRIP achieves efficient and accurate classification by predicting low-resolution regions of interest (ROI), reducing the computational cost of ROI prediction using the low information density of sparse events. The main contributions of this paper are: 1. Designing a hardware-efficient TRIP framework that can automatically generate ROIs, reducing computation and memory costs while maintaining accuracy. 2. Introducing hardware optimizations for TRIP, including the use of differentiable truncated Gaussian kernels (tGK) for ROI generation and implementing hardware optimization algorithms on the SENICA neuromorphic processor. 3. Evaluation on multiple event-based classification datasets demonstrates that the TRIP method significantly reduces computational costs, latency, and energy consumption while maintaining or improving accuracy, showing significant advantages compared to existing solutions. In summary, the objective of this paper is to address the efficiency and cost issues of neuromorphic processors in processing high-resolution event-based visual data. By implementing hardware-optimized ROI prediction through the TRIP framework, overall processing efficiency and energy efficiency are improved.

TRIP: Trainable Region-of-Interest Prediction for Hardware-Efficient Neuromorphic Processing on Event-based Vision

A Hybrid Neuromorphic Object Tracking and Classification Framework for Real-time Systems

Optimizing event-based neural networks on digital neuromorphic architecture: a comprehensive design space exploration

A low-power end-to-end hybrid neuromorphic framework for surveillance applications

Low-power event-based face detection with asynchronous neuromorphic hardware

A neuromorphic event data interpretation approach with hardware reservoir

A High-speed Event-based Object Tracking with Heterogeneous Computing Hardware.

RN-Net: Reservoir Nodes-Enabled Neuromorphic Vision Sensing Network

Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware

REMOT: A Hardware-Software Architecture for Attention-Guided Multi-Object Tracking with Dynamic Vision Sensors on FPGAs

TripleBrain: An Edge Neuromorphic Architecture for High-accuracy Single-layer Spiking Neural Network with On-chip Self-organizing and Reinforcement Learning

TripleBrain: A Compact Neuromorphic Hardware Core With Fast On-Chip Self-Organizing and Reinforcement Spike-Timing Dependent Plasticity

A neuromorphic hardware architecture using the Neural Engineering Framework for pattern recognition

Real-Time Multi-Task Facial Analytics With Event Cameras

Edge AI without Compromise: Efficient, Versatile and Accurate Neurocomputing in Resistive Random-Access Memory

An 1.38nj/inference Clock-Free Mixed-Signal Neuromorphic Architecture Using ReL-PSP Function and Computing-in-Memory.

Detection of Fast-Moving Objects with Neuromorphic Hardware

An Edge Neuromorphic Hardware With Fast On-Chip Error-Triggered Learning on Compressive Sensed Spikes

Random resistive memory-based deep extreme point learning machine for unified visual processing

Ev-Edge: Efficient Execution of Event-based Vision Algorithms on Commodity Edge Platforms

Moving Object Detection and Tracking by Event Frame from Neuromorphic Vision Sensors