Abstract:Event cameras, unlike traditional frame-based cameras, excel in detecting and reporting changes in light intensity on a per-pixel basis. This unique technology offers numerous advantages, including high temporal resolution, low latency, wide dynamic range, and reduced power consumption. These characteristics make event cameras particularly well-suited for sensing applications such as monitoring drivers or human behavior. This paper presents a feasibility study on the using a multitask neural network with event cameras for real-time facial analytics. Our proposed network simultaneously estimates head pose, eye gaze, and facial occlusions. Notably, the network is trained on synthetic event camera data, and its performance is demonstrated and validated using real event data in real-time driving scenarios. To compensate for global head motion, we introduce a novel event integration method capable of handling both short and long-term temporal dependencies. The performance of our facial analytics method is quantitatively evaluated in both controlled lab environments and unconstrained driving scenarios. The results demonstrate that useful accuracy and computational speed is achieved by the proposed method to determining head pose and relative eye-gaze direction. This shows that neuromorphic facial analytics can be realized in real-time and are well-suited for edge/embedded computing deployments. While the improvement ratio in comparison to existing literature may not be as favorable due to the unique event-based vision approach employed, it is crucial to note that our research focuses specifically on event-based vision, which offers distinct advantages over traditional RGB vision. Overall, this study contributes to the emerging field of event-based vision systems and highlights the potential of multitask neural networks combined with event cameras for real-time sensing of human subjects. These techniques can be applied in practical applications such as driver monitoring s- stems, interactive human-computer systems and for human behavior analysis.

Retina-Inspired Object Motion Segmentation for Event-Cameras

Object Motion Sensitivity: A Bio-inspired Solution to the Ego-motion Problem for Event-based Cameras

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

Hardware-Algorithm Re-engineering of Retinal Circuit for Intelligent Object Motion Segmentation

Motion Segmentation for Neuromorphic Aerial Surveillance

Moving Object Detection and Tracking by Event Frame from Neuromorphic Vision Sensors

End-to-End Learning of Object Motion Estimation from Retinal Events for Event-Based Object Tracking

Retinomorphic Object Detection in Asynchronous Visual Streams.

Microsaccade-inspired Event Camera for Robotics

Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes

Real-Time Multi-Task Facial Analytics With Event Cameras

Un-EvMoSeg: Unsupervised Event-based Independent Motion Segmentation

Event-based Real-time Moving Object Detection Based on IMU Ego-motion Compensation.

Eventmd: High-Speed Moving Object Detection Based on Event-Based Video Frames

Retinomorphic Sensing

Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware

Evaluating Image-Based Face and Eye Tracking with Event Cameras

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception.

Real-time face & eye tracking and blink detection using event cameras

Event-Based Motion Segmentation with Spatio-Temporal Graph Cuts.