Abstract:Event cameras, as an emerging imaging technology, offer distinct advantages over traditional RGB cameras, including reduced energy consumption and higher frame rates. However, the limited quantity of available event data presents a significant challenge, hindering their broader development. To alleviate this issue, we introduce a tailored U-shaped State Space Model Knowledge Transfer (USKT) framework for Event-to-RGB knowledge transfer. This framework generates inputs compatible with RGB frames, enabling event data to effectively reuse pre-trained RGB models and achieve competitive performance with minimal parameter tuning. Within the USKT architecture, we also propose a bidirectional reverse state space model. Unlike conventional bidirectional scanning mechanisms, the proposed Bidirectional Reverse State Space Model (BiR-SSM) leverages a shared weight strategy, which facilitates efficient modeling while conserving computational resources. In terms of effectiveness, integrating USKT with ResNet50 as the backbone improves model performance by 0.95%, 3.57%, and 2.9% on DVS128 Gesture, N-Caltech101, and CIFAR-10-DVS datasets, respectively, underscoring USKT's adaptability and effectiveness. The code will be made available upon acceptance.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is **the difficulty in model training and performance improvement caused by the scarcity of Event Camera data**. Specifically, as an emerging imaging technology, the Event Camera has the advantages of low energy consumption and high frame rate, but its limited data volume seriously hinders its broader development and application. To alleviate this problem, the author proposes a knowledge transfer framework specifically for Event Camera data - U - shaped State - Space Model Knowledge Transfer (USKT), to achieve knowledge transfer from event data to RGB data. ### Specific description of the problem: 1. **Advantages and challenges of the Event Camera**: - **Advantages**: The Event Camera can capture pixel - level luminance changes, providing extremely high temporal resolution and low latency, and is especially suitable for capturing fast - moving activities and handling high - dynamic - range scenes. - **Challenges**: The data recording of the Event Camera only occurs when the pixel luminance changes, resulting in a data stream that is usually sparse in visual content, and the feature distribution is quite different from that of RGB images. In addition, the scarcity of Event Camera data makes effective knowledge transfer difficult. 2. **Limitations of existing methods**: - **Domain adaptation methods**: It is difficult to handle the problem of domain mismatch. - **Generative methods**: Although it can simulate sparse event streams to generate additional synthetic RGB data, it consumes a large amount of computing resources. ### Solution: The author proposes a generative U - shaped State - Space Model Knowledge Transfer (USKT) framework, which aims to solve the above problems in the following ways: - **Generate compatible RGB input**: The USKT framework generates input compatible with RGB frames, enabling event data to effectively reuse pre - trained RGB models, thus achieving competitive performance with minimal parameter adjustment. - **Bidirectional Inverse State - Space Model (BiR - SSM)**: A bidirectional inverse state - space model with a shared - weight strategy is introduced, which not only improves the modeling efficiency but also saves computing resources. - **Mixed loss function**: Combine the reconstruction loss and the classification loss to cooperatively optimize the model performance. ### Experimental results: Verified by experiments on multiple datasets such as DVS128 Gesture, N - Caltech101, and CIFAR - 10 - DVS, the USKT framework significantly improves the model performance, increasing the accuracy by 0.95%, 3.57%, and 2.9% respectively, proving its effectiveness and adaptability. ### Summary: By designing the USKT framework, this paper successfully solves the problem of the scarcity of Event Camera data, providing new ideas and technical support for the application and development of Event Cameras.

Event USKT : U-State Space Model in Knowledge Transfer for Event Cameras

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

Graph-based Asynchronous Event Processing for Rapid Object Recognition

SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

State Space Models for Event Cameras

Event Camera Data Pre-training

EventAug: Multifaceted Spatio-Temporal Data Augmentation Methods for Event-based Learning

Joint Framework for Single Image Reconstruction and Super-Resolution with an Event Camera

Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba

Transferring Deep Object and Scene Representations for Event Recognition in Still Images.

Spiking Transfer Learning From RGB Image to Neuromorphic Event Stream

Space-Time Event Clouds for Gesture Recognition: From RGB Cameras to Event Cameras

Event Stream Super-Resolution Via Spatiotemporal Constraint Learning

Knowledge Synergy Learning for Multi-Modal Tracking

A Residual Learning Approach to Deblur and Generate High Frame Rate Video with an Event Camera

Enhancing Video Event Recognition Using Automatically Constructed Semantic-Visual Knowledge Base.

K-LITE: Learning Transferable Visual Models with External Knowledge

IncEventGS: Pose-Free Gaussian Splatting from a Single Event Camera

Path-adaptive Spatio-Temporal State Space Model for Event-based Recognition with Arbitrary Duration

E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning

Learning to Deblur and Generate High Frame Rate Video with an Event Camera