Abstract:Event cameras are bio-inspired sensors that capture the intensity changes asynchronously and output event streams with distinct advantages, such as high temporal resolution. To exploit event cameras for object/action recognition, existing methods predominantly sample and aggregate events in a second-level duration at every fixed temporal interval (or frequency). However, they often face difficulties in capturing the spatiotemporal relationships for longer, e.g., minute-level, events and generalizing across varying temporal frequencies. To fill the gap, we present a novel framework, dubbed PAST-SSM, exhibiting superior capacity in recognizing events with arbitrary duration (e.g., 0.1s to 4.5s) and generalizing to varying inference frequencies. Our key insight is to learn the spatiotemporal relationships from the encoded event features via the state space model (SSM) -- whose linear complexity makes it ideal for modeling high temporal resolution events with longer sequences. To achieve this goal, we first propose a Path-Adaptive Event Aggregation and Scan (PEAS) module to encode events of varying duration into features with fixed dimensions by adaptively scanning and selecting aggregated event frames. On top of PEAS, we introduce a novel Multi-faceted Selection Guiding (MSG) loss to minimize the randomness and redundancy of the encoded features. This subtly enhances the model generalization across different inference frequencies. Lastly, the SSM is employed to better learn the spatiotemporal properties from the encoded features. Moreover, we build a minute-level event-based recognition dataset, named ArDVS100, with arbitrary duration for the benefit of the community. Extensive experiments prove that our method outperforms prior arts by +3.45%, +0.38% and +8.31% on the DVS Action, SeAct and HARDVS datasets, respectively.

Multi-scale Harmonic Mean Time Surfaces for Event-based Object Classification

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

Multilevel Spatial-Temporal Feature Aggregation for Video Object Detection

An Event-based Feature Representation Method for Event Stream Classification Using Deep Spiking Neural Networks

An Event-Driven Object Recognition Model Using Activated Connected Domain Detection

An Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning

Event Stream Learning Using Spatio-Temporal Event Surface

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

A Multi-scale Multivariate Time Series Classification Method Based on Bag of Patterns

A Time-Related Voxel Representation Method for Event Camera

MVT: Multi-Vision Transformer for Event-Based Small Target Detection

Event camera object recognition using spatiotemporal event time surface and reward-modulated spike-timing-dependent plasticity learning rule

Voxel-based Multi-scale Transformer Network for Event Stream Processing

Deep Representation Via Convolutional Neural Network for Classification of Spatiotemporal Event Streams

Transforming Multidimensional Time Series into Interpretable Event Sequences for Advanced Data Mining

TV-Net: Temporal-Variable feature harmonizing Network for multivariate time series classification and interpretation

Temporal-wise Attention Spiking Neural Networks for Event Streams Classification

Multi-Target Multi-Camera Tracking with Human Body Part Semantic Features

Path-adaptive Spatio-Temporal State Space Model for Event-based Recognition with Arbitrary Duration

VMV-GCN: Volumetric Multi-View Based Graph CNN for Event Stream Classification

ECSNet: Spatio-Temporal Feature Learning for Event Camera