Abstract:Event camera-based pattern recognition is a newly arising research topic in recent years. Current researchers usually transform the event streams into images, graphs, or voxels, and adopt deep neural networks for event-based classification. Although good performance can be achieved on simple event recognition datasets, however, their results may be still limited due to the following two issues. Firstly, they adopt spatial sparse event streams for recognition only, which may fail to capture the color and detailed texture information well. Secondly, they adopt either Spiking Neural Networks (SNN) for energy-efficient recognition with suboptimal results, or Artificial Neural Networks (ANN) for energy-intensive, high-performance recognition. However, seldom of them consider achieving a balance between these two aspects. In this paper, we formally propose to recognize patterns by fusing RGB frames and event streams simultaneously and propose a new RGB frame-event recognition framework to address the aforementioned issues. The proposed method contains four main modules, i.e., memory support Transformer network for RGB frame encoding, spiking neural network for raw event stream encoding, multi-modal bottleneck fusion module for RGB-Event feature aggregation, and prediction head. Due to the scarce of RGB-Event based classification dataset, we also propose a large-scale PokerEvent dataset which contains 114 classes, and 27102 frame-event pairs recorded using a DVS346 event camera. Extensive experiments on two RGB-Event based classification datasets fully validated the effectiveness of our proposed framework. We hope this work will boost the development of pattern recognition by fusing RGB frames and event streams. Both our dataset and source code of this work will be released at <a class="link-external link-https" href="https://github.com/Event-AHU/SSTFormer" rel="external noopener nofollow">this https URL</a>.

Bina-Rep Event Frames: a Simple and Effective Representation for Event-based cameras

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

Graph-based Asynchronous Event Processing for Rapid Object Recognition

Learn to See by Events: Color Frame Synthesis from Event and RGB Cameras

Revisiting Event-based Video Frame Interpolation

EvRepSL: Event-Stream Representation via Self-Supervised Learning for Event-Based Vision

A Review of Image Reconstruction Based on Event Cameras

Event-based Continuous Color Video Decompression from Single Frames

Token-Based Spatiotemporal Representation of the Events.

Rethinking Video with a Universal Event-Based Representation

Learning to Deblur and Generate High Frame Rate Video with an Event Camera

Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking

Asynchronous "Events" are Better For Motion Estimation

Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera

Continuous-time Intensity Estimation Using Event Cameras

Event Transformer

E2VIDX: improved bridge between conventional vision and bionic vision

SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

Motion Aware Event Representation-Driven Image Deblurring

Image De-occlusion via Event-enhanced Multi-modal Fusion Hybrid Network

A Residual Learning Approach to Deblur and Generate High Frame Rate Video with an Event Camera