Abstract:We propose a real-time hand gesture interface based on combining a stereo pair of biologically inspired event-based dynamic vision sensor (DVS) silicon retinas with neuromorphic event-driven postprocessing. Compared with conventional vision or 3-D sensors, the use of DVSs, which output asynchronous and sparse events in response to motion, eliminates the need to extract movements from sequences of video frames, and allows significantly faster and more energy-efficient processing. In addition, the rate of input events depends on the observed movements, and thus provides an additional cue for solving the gesture spotting problem, i.e., finding the onsets and offsets of gestures. We propose a postprocessing framework based on spiking neural networks that can process the events received from the DVSs in real time, and provides an architecture for future implementation in neuromorphic hardware devices. The motion trajectories of moving hands are detected by spatiotemporally correlating the stereoscopically verged asynchronous events from the DVSs by using leaky integrate-and-fire (LIF) neurons. Adaptive thresholds of the LIF neurons achieve the segmentation of trajectories, which are then translated into discrete and finite feature vectors. The feature vectors are classified with hidden Markov models, using a separate Gaussian mixture model for spotting irrelevant transition gestures. The disparity information from stereovision is used to adapt LIF neuron parameters to achieve recognition invariant of the distance of the user to the sensor, and also helps to filter out movements in the background of the user. Exploiting the high dynamic range of DVSs, furthermore, allows gesture recognition over a 60-dB range of scene illuminance. The system achieves recognition rates well over 90% under a variety of variable conditions with static and dynamic backgrounds with naïve users.

A Near-Eye DVS-Based End-to-End Eye Tracking Processor for AR/VR Applications

Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware

P‐44: A Smart Eye Tracking System Based on Linear Array Sensors

Foveated Light-Field Display and Real-Time Rendering for Virtual Reality

Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas

Exploring Eye Adaptation in Head-Mounted Display for Energy Efficient Smartphone Virtual Reality.

35‐4: Invited Paper: Next Generation Eye‐tracking Technology for AR/VR Devices

Co-designing a Sub-millisecond Latency Event-based Eye Tracking System with Submanifold Sparse CNN

An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors

Towards Energy Efficient Mobile Eye Tracking for AR Glasses through Optical Sensor Technology

Real-Time Gaze Tracking with Event-Driven Eye Segmentation

Soft, wireless periocular wearable electronics for real-time detection of eye vergence in a virtual reality toward mobile eye therapies

Eye-Tracking Monitoring Based on PMUT Arrays

BlissCam: Boosting Eye Tracking Efficiency with Learned In-Sensor Sparse Sampling

A1920 × 1080 25FPS, 2.4TOPS/W Unified Optical Flow and Depth 6D Vision Processor for Energy-Efficient, Low Power Autonomous Navigation

A 0.99-to-4.38 Uj/class Event-Driven Hybrid Neural Network Processor for Full-Spectrum Neural Signal Analyses

Predictive Temporal Attention on Event-based Video Stream for Energy-efficient Situation Awareness

A power-aware vision-based virtual sensor for real-time edge computing

Toward High Performance, Programmable Extreme-Edge Intelligence for Neuromorphic Vision Sensors utilizing Magnetic Domain Wall Motion-based MTJ

iShadow: Design of a Wearable, Real-Time Mobile Gaze Tracker

Design and Implementation of High-Performance Video Processor for Head-Mounted Displays