Abstract:This paper subverts the traditional understanding of Photoplethysmography (PPG) and opens up a new direction of the utility of PPG in commodity wearable devices, especially in the domain of human computer interaction of fine-grained gesture recognition. We demonstrate that it is possible to leverage the widely deployed PPG sensors in wrist-worn wearable devices to enable finger-level gesture recognition, which could facilitate many emerging human-computer interactions (e.g., sign-language interpretation and virtual reality). While prior solutions in gesture recognition require dedicated devices (e.g., video cameras or IR sensors) or leverage various signals in the environments (e.g., sound, RF or ambient light), this paper introduces the first PPG-based gesture recognition system that can differentiate fine-grained hand gestures at finger level using commodity wearables. Our innovative system harnesses the unique blood flow changes in a user's wrist area to distinguish the user's finger and hand movements. The insight is that hand gestures involve a series of muscle and tendon movements that compress the arterial geometry with different degrees, resulting in significant motion artifacts to the blood flow with different intensity and time duration. By leveraging the unique characteristics of the motion artifacts to PPG, our system can accurately extract the gesture-related signals from the significant background noise (i.e., pulses), and identify different minute finger-level gestures. Extensive experiments are conducted with over 3600 gestures collected from 10 adults. Our prototype study using two commodity PPG sensors can differentiate nine finger-level gestures from American Sign Language with an average recognition accuracy over 88%, suggesting that our PPG-based finger-level gesture recognition system is promising to be one of the most critical components in sign language translation using wearables.

PPG-based Finger-level Gesture Recognition Leveraging Wearables.

G-PPG: A Gesture-related PPG-based Two-Factor Authentication for Wearable Devices.

M-PPG: A Non-intrusive Authentication for Wearable Devices Via PPG Sensors with Motor Excitations

Stretchable e-Skin Patch for Gesture Recognition on the Back of the Hand

Research on Dynamic Gesture Recognition with Low-Pixel ToF-Sensors

Comparative Study of Gesture Recognition Based on Accelerometer and Photoplethysmography Sensor for Gesture Interactions in Wearable Devices

Principal component analysis of photoplethysmography signals for improved gesture recognition

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

Finger Angle-Based Hand Gesture Recognition for Smart Infrastructure Using Wearable Wrist-Worn Camera

Hand Gesture Recognition by a MMG-Based Wearable Device

A Wearable Gesture Recognition System with Ultra-high Accuracy and Robustness Enabled by the Synergy of Multiple Fabric Sensing Devices

Plastic Optical Fiber Enabled Smart Glove for Machine Learning-Based Gesture Recognition

Hand Gesture Recognition using Deep Feature Fusion Network based on Wearable Sensors

Gesture recognition by instantaneous surface EMG images

CAPG-MYO - A Muscle-Computer Interface Supporting User-defined Gesture Recognition.

Ultigesture: A Wristband-based Platform for Continuous Gesture Control in Healthcare

Gesture Recognition Framework for Teleoperation of Infrared (IR) Consumer Devices Using a Novel pFMG Soft Armband

Exploration on the Negative Effects of Sensor Shifts in Photoplethysmography-based Gesture Recognition and a Solution based on Transfer Learning

Gesture-Enabled Remote Control for Healthcare

Self‐Powered Gesture Recognition Wristband Enabled by Machine Learning for Full Keyboard and Multicommand Input

Flexible Non-contact Capacitive Sensing for Hand Gesture Recognition