Abstract:Event camera-based pattern recognition is a newly arising research topic in recent years. Current researchers usually transform the event streams into images, graphs, or voxels, and adopt deep neural networks for event-based classification. Although good performance can be achieved on simple event recognition datasets, however, their results may be still limited due to the following two issues. Firstly, they adopt spatial sparse event streams for recognition only, which may fail to capture the color and detailed texture information well. Secondly, they adopt either Spiking Neural Networks (SNN) for energy-efficient recognition with suboptimal results, or Artificial Neural Networks (ANN) for energy-intensive, high-performance recognition. However, seldom of them consider achieving a balance between these two aspects. In this paper, we formally propose to recognize patterns by fusing RGB frames and event streams simultaneously and propose a new RGB frame-event recognition framework to address the aforementioned issues. The proposed method contains four main modules, i.e., memory support Transformer network for RGB frame encoding, spiking neural network for raw event stream encoding, multi-modal bottleneck fusion module for RGB-Event feature aggregation, and prediction head. Due to the scarce of RGB-Event based classification dataset, we also propose a large-scale PokerEvent dataset which contains 114 classes, and 27102 frame-event pairs recorded using a DVS346 event camera. Extensive experiments on two RGB-Event based classification datasets fully validated the effectiveness of our proposed framework. We hope this work will boost the development of pattern recognition by fusing RGB frames and event streams. Both our dataset and source code of this work will be released at <a class="link-external link-https" href="https://github.com/Event-AHU/SSTFormer" rel="external noopener nofollow">this https URL</a>.

Space-Time Event Clouds for Gesture Recognition: From RGB Cameras to Event Cameras

Rethinking Human Pose Estimation for Autonomous Driving with 3D Event Representations.

Graph-based Asynchronous Event Processing for Rapid Object Recognition

Research on Dynamic Gesture Recognition with Low-Pixel ToF-Sensors

Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba

A Cascaded Timestamp-Free Event Camera Image Compression Method for Gesture Recognition

TTPOINT: A Tensorized Point Cloud Network for Lightweight Action Recognition with Event Cameras

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

Event camera object recognition using spatiotemporal event time surface and reward-modulated spike-timing-dependent plasticity learning rule

Real-Time Hand Gesture Recognition Using RGB-D Sensor

EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

Backlight and dim space object detection based on a novel event camera

Spatiotemporal Learning of Dynamic Gestures from 3D Point Cloud Data

Adaptive spatio-temporal clustering for small object detection using event-based camera

Duodepth: Static Gesture Recognition Via Dual Depth Sensors

EventPoint: Self-Supervised Interest Point Detection and Description for Event-based Camera

Denoising Method Based on Salient Region Recognition for the Spatiotemporal Event Stream

SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition

Event-Based Gesture Recognition With Dynamic Background Suppression Using Smartphone Computational Capabilities

IncEventGS: Pose-Free Gaussian Splatting from a Single Event Camera