Abstract:Face analysis has been studied from different angles to infer emotion, poses, shapes, and landmarks. Traditionally RGB cameras are used, yet for fine-grained tasks standard sensors might not be up to the task due to their latency, making it impossible to record and detect micro-movements that carry a highly informative signal, which is necessary for inferring the true emotions of a subject. Event cameras have been increasingly gaining interest as a possible solution to this and similar high-frame rate tasks. We propose a novel spatiotemporal Vision Transformer model that uses Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA) to enhance the accuracy of Action Unit classification from event streams. We also address the lack of labeled event data in the literature, which can be considered one of the main causes of an existing gap between the maturity of RGB and neuromorphic vision models. Gathering data is harder in the event domain since it cannot be crawled from the web and labeling frames should take into account event aggregation rates and the fact that static parts might not be visible in certain frames. To this end, we present FACEMORPHIC, a temporally synchronized multimodal face dataset composed of RGB videos and event streams. The dataset is annotated at a video level with facial Action Units and contains streams collected with various possible applications, ranging from 3D shape estimation to lip-reading. We then show how temporal synchronization can allow effective neuromorphic face analysis without the need to manually annotate videos: we instead leverage cross-modal supervision bridging the domain gap by representing face shapes in a 3D space. Our proposed model outperforms baseline methods by effectively capturing spatial and temporal information, crucial for recognizing subtle facial micro-expressions.

STformer: Spatial-Temporal Transformer for Early Warning of Unplanned Extubation in ICU

Spatio-temporal features for fast early warning of unplanned self-extubation in ICU

PLFormer: Prompt Learning for Early Warning of Unplanned Extubation in ICU.

Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

Unformer: A Transformer-Based Approach for Adaptive Multi-Scale Feature Aggregation in Underwater Image Enhancement

STFormer: Spatial-Temporal-Aware Transformer for Video Instance Segmentation

ISTVT: Interpretable Spatial-Temporal Video Transformer for Deepfake Detection

UTRAD: Anomaly detection and localization with U-Transformer

ESTformer: Transformer Utilizing Spatiotemporal Dependencies for EEG Super-resolution

Learning Complementary Spatial-Temporal Transformer for Video Salient Object Detection

UniST: Towards Unifying Saliency Transformer for Video Saliency Prediction and Detection

Long Short-Term Transformer for Online Action Detection

LAST: LAtent Space-Constrained Transformers for Automatic Surgical Phase Recognition and Tool Presence Detection.

Spatio-temporal Transformers for Action Unit Classification with Event Cameras

Exploring reliable infrared object tracking with spatio-temporal fusion transformer

STAU: A SpatioTemporal-Aware Unit for Video Prediction and Beyond

Surveillance Camera-based Cardio-respiratory Monitoring for Critical Patients in ICU

An End-to-End Spatial-Temporal Transformer Model for Surgical Action Triplet Recognition

ST-former for short-term passenger flow prediction during COVID-19 in urban rail transit system

STST: Spatial-Temporal Specialized Transformer for Skeleton-based Action Recognition