Abstract:Event cameras are neuromorphic vision sensors that record a scene as sparse and asynchronous event streams. Most event-based methods project events into dense frames and process them using conventional vision models, resulting in high computational complexity. A recent trend is to develop point-based networks that achieve efficient event processing by learning sparse representations. However, existing works may lack robust local information aggregators and effective feature interaction operations, thus limiting their modeling capabilities. To this end, we propose an attention-aware model named Event Voxel Set Transformer (EVSTr) for efficient spatiotemporal representation learning on event streams. It first converts the event stream into voxel sets and then hierarchically aggregates voxel features to obtain robust representations. The core of EVSTr is an event voxel transformer encoder that consists of two well-designed components, including the Multi-Scale Neighbor Embedding Layer (MNEL) for local information aggregation and the Voxel Self-Attention Layer (VSAL) for global feature interaction. Enabling the network to incorporate a long-range temporal structure, we introduce a segment modeling strategy (S$^{2}$TM) to learn motion patterns from a sequence of segmented voxel sets. The proposed model is evaluated on two recognition tasks, including object classification and action recognition. To provide a convincing model evaluation, we present a new event-based action recognition dataset (NeuroHAR) recorded in challenging scenarios. Comprehensive experiments show that EVSTr achieves state-of-the-art performance while maintaining low model complexity.

Event Stream Learning Using Spatio-Temporal Event Surface

An Event-based Feature Representation Method for Event Stream Classification Using Deep Spiking Neural Networks

Event-driven Spiking Neural Networks with Spike-Based Learning

Event Stream Super-Resolution Via Spatiotemporal Constraint Learning

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

Spiking Neural Network Recognition Method Based on Dynamic Visual Motion Features

Deep Representation Via Convolutional Neural Network for Classification of Spatiotemporal Event Streams

Event camera object recognition using spatiotemporal event time surface and reward-modulated spike-timing-dependent plasticity learning rule

Multi-scale Harmonic Mean Time Surfaces for Event-based Object Classification

[A bio-inspired hierarchical spiking neural network with biological synaptic plasticity for event camera object recognition].

Representation Learning using Event-based STDP

An Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning

Effective AER Object Classification Using Segmented Probability-Maximization Learning in Spiking Neural Networks

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

Bio-inspired Categorization Using Event-Driven Feature Extraction and Spike-Based Learning.

Representation Learning on Event Stream via an Elastic Net-incorporated Tensor Network

Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams

EventPoint: Self-Supervised Interest Point Detection and Description for Event-based Camera

STCA: Spatio-Temporal Credit Assignment with Delayed Feedback in Deep Spiking Neural Networks

Learning Local Event-based Descriptor for Patch-based Stereo Matching

Spatiotemporal Feature Learning for Event-Based Vision