Abstract:Event cameras, unlike traditional frame-based cameras, excel in detecting and reporting changes in light intensity on a per-pixel basis. This unique technology offers numerous advantages, including high temporal resolution, low latency, wide dynamic range, and reduced power consumption. These characteristics make event cameras particularly well-suited for sensing applications such as monitoring drivers or human behavior. This paper presents a feasibility study on the using a multitask neural network with event cameras for real-time facial analytics. Our proposed network simultaneously estimates head pose, eye gaze, and facial occlusions. Notably, the network is trained on synthetic event camera data, and its performance is demonstrated and validated using real event data in real-time driving scenarios. To compensate for global head motion, we introduce a novel event integration method capable of handling both short and long-term temporal dependencies. The performance of our facial analytics method is quantitatively evaluated in both controlled lab environments and unconstrained driving scenarios. The results demonstrate that useful accuracy and computational speed is achieved by the proposed method to determining head pose and relative eye-gaze direction. This shows that neuromorphic facial analytics can be realized in real-time and are well-suited for edge/embedded computing deployments. While the improvement ratio in comparison to existing literature may not be as favorable due to the unique event-based vision approach employed, it is crucial to note that our research focuses specifically on event-based vision, which offers distinct advantages over traditional RGB vision. Overall, this study contributes to the emerging field of event-based vision systems and highlights the potential of multitask neural networks combined with event cameras for real-time sensing of human subjects. These techniques can be applied in practical applications such as driver monitoring s- stems, interactive human-computer systems and for human behavior analysis.

Integrated Multi-Scale Event Verification In An Augmented Foreground Motion Space

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

Local Fast R-Cnn Flow For Object-Centric Event Recognition In Complex Traffic Scenes

Event-Based Vision Enhanced: A Joint Detection Framework in Autonomous Driving

Traffic Sign Detection With Event Cameras and DCNN

Vehicle Behavior Recognition using Multi-Stream 3D Convolutional Neural Network

Multi-person Multi-Camera Tracking for Live Stream Videos Based on Improved Motion Model and Matching Cascade

Multi-scale Harmonic Mean Time Surfaces for Event-based Object Classification

VMV-GCN: Volumetric Multi-View Based Graph CNN for Event Stream Classification

DevNet: A Deep Event Network for Multimedia Event Detection and Evidence Recounting

An automatic verification method for vehicle line-pressing violation based on CNN and geometric projection

Security Event Recognition for Visual Surveillance

Real-Time Multi-Task Facial Analytics With Event Cameras

Real-World Spatial Synchronization of Event-CMOS Cameras through Deep Learning: A Novel CNN-DGCNN Approach

Scalable Surrogate Verification of Image-based Neural Network Control Systems using Composition and Unrolling

Video Surveillance of Highway Traffic Events by Deep Learning Architectures

Multi-object Events Recognition from Video Sequences Using Extended Finite State Machine

MVF-Net: A Multi-view Fusion Network for Event-based Object Classification

Multi-Dimensional Traffic Congestion Detection Based on Fusion of Visual Features and Convolutional Neural Network

Actor-Multi-Scale Context Bidirectional Higher Order Interactive Relation Network for Spatial-Temporal Action Localization.

Event detection in coarsely annotated sports videos via parallel multi receptive field 1D convolutions