Abstract:Event cameras capture visual information with a high temporal resolution and a wide dynamic range. This enables capturing visual information at fine time granularities (e.g., microseconds) in rapidly changing environments. This makes event cameras highly useful for high-speed robotics tasks involving rapid motion, such as high-speed perception, object tracking, and control. However, convolutional neural network inference on event camera streams cannot currently perform real-time inference at the high speeds at which event cameras operate - current CNN inference times are typically closer in order of magnitude to the frame rates of regular frame-based cameras. Real-time inference at event camera rates is necessary to fully leverage the high frequency and high temporal resolution that event cameras offer. This paper presents EvConv, a new approach to enable fast inference on CNNs for inputs from event cameras. We observe that consecutive inputs to the CNN from an event camera have only small differences between them. Thus, we propose to perform inference on the difference between consecutive input tensors, or the increment. This enables a significant reduction in the number of floating-point operations required (and thus the inference latency) because increments are very sparse. We design EvConv to leverage the irregular sparsity in increments from event cameras and to retain the sparsity of these increments across all layers of the network. We demonstrate a reduction in the number of floating operations required in the forward pass by up to 98%. We also demonstrate a speedup of up to 1.6X for inference using CNNs for tasks such as depth estimation, object recognition, and optical flow estimation, with almost no loss in accuracy.

Compressed Event Sensing (CES) Volumes for Event Cameras

Learning-based Lossless Event Data Compression

ECSNet: Spatio-Temporal Feature Learning for Event Camera

Event Enhanced High-Quality Image Recovery

SCSE-E2VID: Improved event-based video reconstruction with an event camera.

Improved Event-Based Dense Depth Estimation Via Optical Flow Compensation.

Time Information Integration Network for Event Cameras

A Cascaded Timestamp-Free Event Camera Image Compression Method for Gesture Recognition

VMV-GCN: Volumetric Multi-View Based Graph CNN for Event Stream Classification

Flow-Based Visual Stream Compression for Event Cameras

Event-Stream Super Resolution using Sigma-Delta Neural Network

Event-based Video Reconstruction Using Transformer

EDeNN: Event Decay Neural Networks for low latency vision

Double Deep Learning-based Event Data Coding and Classification

Deep Representation Via Convolutional Neural Network for Classification of Spatiotemporal Event Streams

Low Complexity Learning-based Lossless Event-based Compression

Time-Ordered Recent Event (TORE) Volumes for Event Cameras

EvConv: Fast CNN Inference on Event Camera Inputs For High-Speed Robot Perception

V2CE: Video to Continuous Events Simulator

NeRFE: Free View Synthesis for Event Data

E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning