Abstract:Event camera-based pattern recognition is a newly arising research topic in recent years. Current researchers usually transform the event streams into images, graphs, or voxels, and adopt deep neural networks for event-based classification. Although good performance can be achieved on simple event recognition datasets, however, their results may be still limited due to the following two issues. Firstly, they adopt spatial sparse event streams for recognition only, which may fail to capture the color and detailed texture information well. Secondly, they adopt either Spiking Neural Networks (SNN) for energy-efficient recognition with suboptimal results, or Artificial Neural Networks (ANN) for energy-intensive, high-performance recognition. However, seldom of them consider achieving a balance between these two aspects. In this paper, we formally propose to recognize patterns by fusing RGB frames and event streams simultaneously and propose a new RGB frame-event recognition framework to address the aforementioned issues. The proposed method contains four main modules, i.e., memory support Transformer network for RGB frame encoding, spiking neural network for raw event stream encoding, multi-modal bottleneck fusion module for RGB-Event feature aggregation, and prediction head. Due to the scarce of RGB-Event based classification dataset, we also propose a large-scale PokerEvent dataset which contains 114 classes, and 27102 frame-event pairs recorded using a DVS346 event camera. Extensive experiments on two RGB-Event based classification datasets fully validated the effectiveness of our proposed framework. We hope this work will boost the development of pattern recognition by fusing RGB frames and event streams. Both our dataset and source code of this work will be released at <a class="link-external link-https" href="https://github.com/Event-AHU/SSTFormer" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper aims to address the following two main issues: 1. **Limitations of Existing Methods**: Current research on pattern recognition based on event cameras typically converts event streams into images, graphs, or voxels and uses deep neural networks for classification. Although good performance can be achieved on simple event recognition datasets, there are still limitations in complex scenarios: - Recognizing using only spatially sparse event streams makes it difficult to capture color and detailed texture information. - When using Spiking Neural Networks (SNN) for low-energy recognition, the results are suboptimal; while using Artificial Neural Networks (ANN) for high-energy, high-performance recognition, the energy consumption is high. Few methods attempt to find a balance between the two. 2. **Insufficiency of Multimodal Fusion**: Existing work is based solely on a single event camera, and the event stream is spatially sparse, making it ineffective in capturing color and detailed texture information crucial for pattern recognition. There are few methods that combine multimodal cues to achieve higher performance. ### Solution Overview To address the above issues, the authors propose a novel framework that fuses RGB frames and event streams for pattern recognition. Specifically, the framework includes the following modules: - **Spiking Neural Network (SNN)**: Directly processes the raw event stream for efficient perception. - **Memory-Supported Transformer Network (MST)**: Encodes the spatiotemporal information of RGB frames. - **Multimodal Bottleneck Fusion Module (MBF)**: Fuses RGB and event features for pattern recognition. - **Prediction Head**: Performs the final pattern classification. Additionally, due to the lack of large-scale RGB-event classification datasets, the authors also propose a new dataset named PokerEvent, which contains 114 categories and 27,102 pairs of RGB-event samples, to validate the effectiveness of the proposed framework. Through this framework and dataset, the authors hope to find a better balance between energy consumption and recognition accuracy.

SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

Unleashing the Power of CNN and Transformer for Balanced RGB-Event Video Recognition

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

An Event-Driven Object Recognition Model Using Activated Connected Domain Detection

Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

An Event-based Feature Representation Method for Event Stream Classification Using Deep Spiking Neural Networks

Spiking Transfer Learning From RGB Image to Neuromorphic Event Stream

An Event-Driven Computational System With Spiking Neurons For Object Recognition

Event camera object recognition using spatiotemporal event time surface and reward-modulated spike-timing-dependent plasticity learning rule

Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition

Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams

An Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

Graph-based Asynchronous Event Processing for Rapid Object Recognition

Multi-scale Harmonic Mean Time Surfaces for Event-based Object Classification

Learning Bottleneck Transformer for Event Image-Voxel Feature Fusion based Classification

Event Stream Super-Resolution Via Spatiotemporal Constraint Learning

Continuous-time Object Segmentation using High Temporal Resolution Event Camera

TENet: Targetness Entanglement Incorporating with Multi-Scale Pooling and Mutually-Guided Fusion for RGB-E Object Tracking

Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking