Abstract:Event-based vision represents a paradigm shift in how vision information is captured and processed. By only responding to dynamic intensity changes in the scene, event-based sensing produces far less data than conventional frame-based cameras, promising to springboard a new generation of high-speed, low-power machines for edge intelligence. However, processing such dynamically sparse input originated from event cameras efficiently in real time, particularly with complex deep neural networks (DNN), remains a formidable challenge. Existing solutions that employ GPUs and other frame-based DNN accelerators often struggle to efficiently process the dynamically sparse event data, missing the opportunities to improve processing efficiency with sparse data. To address this, we propose ESDA, a composable dynamic sparse dataflow architecture that allows customized DNN accelerators to be constructed rapidly on FPGAs for event-based vision tasks. ESDA is a modular system that is composed of a set of parametrizable modules for each network layer type. These modules share a uniform sparse token-feature interface and can be connected easily to compose an all-on-chip dataflow accelerator on FPGA for each network model. To fully exploit the intrinsic sparsity in event data, ESDA incorporates the use of submanifold sparse convolutions that largely enhance the activation sparsity throughout the layers while simplifying hardware implementation. Finally, a network architecture and hardware implementation co-optimizing framework that allows tradeoffs between accuracy and performance is also presented. Experimental results demonstrate that when compared with existing GPU and hardware-accelerated solutions, ESDA achieves substantial speedup and improvement in energy efficiency across different applications, and it allows much wider design space for real-world deployments.

An FPGA Implementation of an Event-Driven Unsupervised Feature Extraction Algorithm for Pattern Recognition

High-performance Subpixel Edge Location Based on FPGA for Horizon Sensors

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

Event-Based Feature Extraction Using Adaptive Selection Thresholds

Implementation of High Performance Hardware Architecture of Face Recognition Algorithm Based on Local Binary Pattern on Fpga

FPGA Implementation of a Hardware Optimized Automatic Target Detection and Classification Algorithm for Hyperspectral Image Analysis

Efficient Implementation of LMS Adaptive Filter based FECG Extraction on an FPGA

FPGA implementation of full HD real-time depth estimation

Face Recognition with Hybrid Efficient Convolution Algorithms on FPGAs

Embedded FPGA developments in 130 nm and 28 nm CMOS for machine learning in particle detector readout

FPGA-based Efficient Implementation of SURF Algorithm

Real-time face detection and lip feature extraction using field-programmable gate arrays

Embedded FPGA Developments in 130nm and 28nm CMOS for Machine Learning in Particle Detector Readout

FPGA-Based Feature Extraction and Tracking Accelerator for Real-Time Visual SLAM

A Composable Dynamic Sparse Dataflow Architecture for Efficient Event-based Vision Processing on FPGA

CNN-based Feature-point Extraction for Real-time Visual SLAM on Embedded FPGA

FPGA Implementation of An Event-driven Saliency-based Selective Attention Model

A Ubiquitous Machine Learning Accelerator With Automatic Parallelization on FPGA

A Fast Face Recognition System Based on Deep Learning

Efficient Edge Detection on Low-Cost FPGAs

On-Board Detection and Matching of Feature Points