Abstract:Recognizing and comprehending human actions and gestures is a crucial perception requirement for robots to interact with humans and carry out tasks in diverse domains, including service robotics, healthcare, and manufacturing. Event cameras, with their ability to capture fast-moving objects at a high temporal resolution, offer new opportunities compared to standard action recognition in RGB videos. However, previous research on event camera action recognition has primarily focused on sensor-specific network architectures and image encoding, which may not be suitable for new sensors and limit the use of recent advancements in transformer-based architectures. In this study, we employ a computationally efficient model, namely the video transformer network (VTN), which initially acquires spatial embeddings per event-frame and then utilizes a temporal self-attention mechanism. In order to better adopt the VTN for the sparse and fine-grained nature of event data, we design Event-Contrastive Loss ($\mathcal{L}_{EC}$) and event-specific augmentations. Proposed $\mathcal{L}_{EC}$ promotes learning fine-grained spatial cues in the spatial backbone of VTN by contrasting temporally misaligned frames. We evaluate our method on real-world action recognition of N-EPIC Kitchens dataset, and achieve state-of-the-art results on both protocols - testing in seen kitchen (\textbf{74.9\%} accuracy) and testing in unseen kitchens (\textbf{42.43\% and 46.66\% Accuracy}). Our approach also takes less computation time compared to competitive prior approaches, which demonstrates the potential of our framework \textit{EventTransAct} for real-world applications of event-camera based action recognition. Project Page: \url{<a class="link-external link-https" href="https://tristandb8.github.io/EventTransAct_webpage/" rel="external noopener nofollow">this https URL</a>}

Event-based Video Reconstruction Using Transformer

Event Transformer

Event-based Monocular Dense Depth Estimation with Recurrent Transformers

Event-based Monocular Depth Estimation with Recurrent Transformers

Unleashing the Power of CNN and Transformer for Balanced RGB-Event Video Recognition

Event Transformer+. A multi-purpose solution for efficient event data processing

Adaptive Vision Transformer for Event-Based Human Pose Estimation

EventTransAct: A video transformer-based framework for Event-camera based action recognition

GET: Group Event Transformer for Event-Based Vision

Learning Adaptive Parameter Representation for Event-based Video Reconstruction

Voxel-based Multi-scale Transformer Network for Event Stream Processing

Learning Bottleneck Transformer for Event Image-Voxel Feature Fusion based Classification

Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams

Learning to Reconstruct High Speed and High Dynamic Range Videos from Events

Sparse-E2VID: A Sparse Convolutional Model for Event-Based Video Reconstruction Trained with Real Event Noise.

Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers

SpikingViT: a Multi-scale Spiking Vision Transformer Model for Event-based Object Detection

Transformer-Based Domain Adaptation for Event Data Classification

Token-Based Spatiotemporal Representation of the Events.

SCSE-E2VID: Improved event-based video reconstruction with an event camera.

Event Enhanced High-Quality Image Recovery