Abstract:Event-based vision represents a paradigm shift in how vision information is captured and processed. By only responding to dynamic intensity changes in the scene, event-based sensing produces far less data than conventional frame-based cameras, promising to springboard a new generation of high-speed, low-power machines for edge intelligence. However, processing such dynamically sparse input originated from event cameras efficiently in real time, particularly with complex deep neural networks (DNN), remains a formidable challenge. Existing solutions that employ GPUs and other frame-based DNN accelerators often struggle to efficiently process the dynamically sparse event data, missing the opportunities to improve processing efficiency with sparse data. To address this, we propose ESDA, a composable dynamic sparse dataflow architecture that allows customized DNN accelerators to be constructed rapidly on FPGAs for event-based vision tasks. ESDA is a modular system that is composed of a set of parametrizable modules for each network layer type. These modules share a uniform sparse token-feature interface and can be connected easily to compose an all-on-chip dataflow accelerator on FPGA for each network model. To fully exploit the intrinsic sparsity in event data, ESDA incorporates the use of submanifold sparse convolutions that largely enhance the activation sparsity throughout the layers while simplifying hardware implementation. Finally, a network architecture and hardware implementation co-optimizing framework that allows tradeoffs between accuracy and performance is also presented. Experimental results demonstrate that when compared with existing GPU and hardware-accelerated solutions, ESDA achieves substantial speedup and improvement in energy efficiency across different applications, and it allows much wider design space for real-world deployments.

High-definition event frame generation using SoC FPGA devices

Live Video Analytics with FPGA-based Smart Cameras

High speed video signal acquisition and processing system based on FPGA technique

FPGA implementation of full HD real-time depth estimation

High-performance Subpixel Edge Location Based on FPGA for Horizon Sensors

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

Full Hd Real-Time Depth Estimation Algorithm and Hardware Implementation

Real-Time CLAHE Algorithm Implementation in SoC FPGA Device for 4K UHD Video Stream

Event-based vision on FPGAs -- a survey

Real-Time Efficient FPGA Implementation of the Multi-Scale Lucas-Kanade and Horn-Schunck Optical Flow Algorithms for a 4K Video Stream

FPGA-Based Real-Time Super-Resolution System for Ultra High Definition Videos

A Real-Time Full HD 2D-to-3d Video Conversion System Based on FPGA

High-performance architecture for real-time high-definition short-wave infrared streaming video processing and its field programmable gate array prototype

Increasing the scalability of graph convolution for FPGA-implemented event-based vision

Full High-Definition Real-Time Depth Estimation for Three-Dimensional Video System

Real-time video processing model for stereoscopic display system based on FPGA

Dynamic Vision Sensor integration on FPGA-based CNN accelerators for high-speed visual classification

A Composable Dynamic Sparse Dataflow Architecture for Efficient Event-based Vision Processing on FPGA

A wide-spectrum and high-speed CMOS system based on FPGA

Winograd-Based Real-Time Super-Resolution System on FPGA

Design of high-speed real-time sensor image processing based on FPGA and DDR3