Abstract:Event cameras are bio-inspired sensors that capture the intensity changes asynchronously and output event streams with distinct advantages, such as high temporal resolution. To exploit event cameras for object/action recognition, existing methods predominantly sample and aggregate events in a second-level duration at every fixed temporal interval (or frequency). However, they often face difficulties in capturing the spatiotemporal relationships for longer, e.g., minute-level, events and generalizing across varying temporal frequencies. To fill the gap, we present a novel framework, dubbed PAST-SSM, exhibiting superior capacity in recognizing events with arbitrary duration (e.g., 0.1s to 4.5s) and generalizing to varying inference frequencies. Our key insight is to learn the spatiotemporal relationships from the encoded event features via the state space model (SSM) -- whose linear complexity makes it ideal for modeling high temporal resolution events with longer sequences. To achieve this goal, we first propose a Path-Adaptive Event Aggregation and Scan (PEAS) module to encode events of varying duration into features with fixed dimensions by adaptively scanning and selecting aggregated event frames. On top of PEAS, we introduce a novel Multi-faceted Selection Guiding (MSG) loss to minimize the randomness and redundancy of the encoded features. This subtly enhances the model generalization across different inference frequencies. Lastly, the SSM is employed to better learn the spatiotemporal properties from the encoded features. Moreover, we build a minute-level event-based recognition dataset, named ArDVS100, with arbitrary duration for the benefit of the community. Extensive experiments prove that our method outperforms prior arts by +3.45%, +0.38% and +8.31% on the DVS Action, SeAct and HARDVS datasets, respectively.

M-LSM: An Improved Multi-Liquid State Machine for Event-Based Vision Recognition

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

LSM: Learning Subspace Minimization for Low-Level Vision

Resistive memory-based zero-shot liquid state machine for multimodal event data learning

Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion

A Novel State Space Model with Local Enhancement and State Sharing for Image Fusion

Deep Fusion of Multiple Semantic Cues for Complex Event Recognition

Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model

An Adaptive Parameter Optimization Deep Learning Model for Energetic Liquid Vision Recognition Based on Feedback Mechanism

MSCS: Multi-stage Feature Learning with Channel-Spatial Attention Mechanism for Infrared and Visible Image Fusion

Path-adaptive Spatio-Temporal State Space Model for Event-based Recognition with Arbitrary Duration

EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality

Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation

Increasing Liquid State Machine Performance with Edge-of-Chaos Dynamics Organized by Astrocyte-modulated Plasticity

Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusion

2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification

Video Action Recognition with Attentive Semantic Units

A Multi-Stage Visible and Infrared Image Fusion Network Based on Attention Mechanism

Instance-aware Image and Sentence Matching with Selective Multimodal LSTM