Abstract:We propose a real-time hand gesture interface based on combining a stereo pair of biologically inspired event-based dynamic vision sensor (DVS) silicon retinas with neuromorphic event-driven postprocessing. Compared with conventional vision or 3-D sensors, the use of DVSs, which output asynchronous and sparse events in response to motion, eliminates the need to extract movements from sequences of video frames, and allows significantly faster and more energy-efficient processing. In addition, the rate of input events depends on the observed movements, and thus provides an additional cue for solving the gesture spotting problem, i.e., finding the onsets and offsets of gestures. We propose a postprocessing framework based on spiking neural networks that can process the events received from the DVSs in real time, and provides an architecture for future implementation in neuromorphic hardware devices. The motion trajectories of moving hands are detected by spatiotemporally correlating the stereoscopically verged asynchronous events from the DVSs by using leaky integrate-and-fire (LIF) neurons. Adaptive thresholds of the LIF neurons achieve the segmentation of trajectories, which are then translated into discrete and finite feature vectors. The feature vectors are classified with hidden Markov models, using a separate Gaussian mixture model for spotting irrelevant transition gestures. The disparity information from stereovision is used to adapt LIF neuron parameters to achieve recognition invariant of the distance of the user to the sensor, and also helps to filter out movements in the background of the user. Exploiting the high dynamic range of DVSs, furthermore, allows gesture recognition over a 60-dB range of scene illuminance. The system achieves recognition rates well over 90% under a variety of variable conditions with static and dynamic backgrounds with naïve users.

Real-time motor rotation frequency detection with event-based visual and spike-based auditory AER sensory integration for FPGA

Event-Based Angular Speed Measurement and Movement Monitoring

Feedforward Categorization on AER Motion Events Using Cortex-Like Features in a Spiking Neural Network

Interfacing PDM MEMS microphones with PFM spiking systems: Application for Neuromorphic Auditory Sensors

Field-programmable gate array implementation of a probabilistic neural network for motor cortical decoding in rats.

Obstacle Avoidance and Target Acquisition for Robot Navigation Using a Mixed Signal Analog/Digital Neuromorphic Processing System

EventF2S: Asynchronous and Sparse Spiking AER Framework using Neuromorphic-Friendly Algorithm

A Digital Hardware System for Spiking Network of Tactile Afferents

SPAIC: A sub-$μ$W/Channel, 16-Channel General-Purpose Event-Based Analog Front-End with Dual-Mode Encoders

A 4096 channel event-based multielectrode array with asynchronous outputs compatible with neuromorphic processors

A Real-Time Embedded System Design for ERD/ERS Measurement on EEG-Based Brain-Computer Interfaces

Adaptive motor control and learning in a spiking neural network realised on a mixed-signal neuromorphic processor

FPGA-Based Implementation of an Event-Driven Spiking Multi-Kernel Convolution Architecture

A High-Speed Low-Cost VLSI System Capable of On-Chip Online Learning for Dynamic Vision Sensor Data Classification

Dynamic Vision Sensor integration on FPGA-based CNN accelerators for high-speed visual classification

Variable Frequency Driver System of Asynchronous Motor Based on FPGA

Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas

Embodied Neuromorphic Vision with Event-Driven Random Backpropagation

An MPSoC-based on-line Edge Infrastructure for Embedded Neuromorphic Robotic Controllers

Processing EMG signals using reservoir computing on an event-based neuromorphic system

Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware