Abstract:Event cameras are bio-inspired dynamic vision sensors that are superior to frame-based cameras in terms of low power consumption, high dynamic range, and high temporal resolution in computer vision tasks. Recent advances in voxel-based representation learning have successfully exploited the sparsity of events with low computational complexity, but face challenges in extracting spatio-temporal features within voxels and representative global dependencies between voxels, thus limiting their representation power. In this work, towards a better trade-off between accuracy and computation overhead, we propose a novel voxel-based multi-scale transformer network (VMST-Net) to process event streams. Specifically, VMST-Net projects events within voxels into multi-channel frames along the time axis, such that 2D convolutions could be leveraged to encode spatio-temporal features in voxels. Then, VMST-Net utilizes a novel multi-scale multi-head self-attention (MSMHSA) mechanism with a multi-scale fusion (MSF) module that allows different heads within each layer to attend different scale 3D neighborhoods to adaptively aggregate the coarse-to-fine voxel features with little computational costs and parameters. Moreover, to model effective global features while saving computations, we aggregate features in a local-to-global manner by enlarging the coverage of 3D neighborhoods as the network gets deeper. Extensive experimental results on benchmark datasets demonstrate that our model advances state-of-the-art accuracy with low model complexity and computational complexity in all three visual tasks, including object classification, action recognition, and human pose estimation.

Deep Representation Via Convolutional Neural Network for Classification of Spatiotemporal Event Streams

An Event-based Feature Representation Method for Event Stream Classification Using Deep Spiking Neural Networks

Event Stream Learning Using Spatio-Temporal Event Surface

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Spiking Neural Network Recognition Method Based on Dynamic Visual Motion Features

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

Event Stream Super-Resolution Via Spatiotemporal Constraint Learning

VMV-GCN: Volumetric Multi-View Based Graph CNN for Event Stream Classification

Multi-scale Harmonic Mean Time Surfaces for Event-based Object Classification

Temporal-wise Attention Spiking Neural Networks for Event Streams Classification

Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams

CIFAR10-DVS: an Event-Stream Dataset for Object Classification

A dynamic vision sensor object recognition model based on trainable event-driven convolution and spiking attention mechanism

Event-Stream Super Resolution using Sigma-Delta Neural Network

[A bio-inspired hierarchical spiking neural network with biological synaptic plasticity for event camera object recognition].

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Representation Learning using Event-based STDP

Voxel-based Multi-scale Transformer Network for Event Stream Processing

Event camera object recognition using spatiotemporal event time surface and reward-modulated spike-timing-dependent plasticity learning rule