Abstract:Since the advent of computing, humans have sought computer input technologies that are expressive, intuitive, and universal. While diverse modalities have been developed, including keyboards, mice, and touchscreens, they require interaction with an intermediary device that can be limiting, especially in mobile scenarios. Gesture-based systems utilize cameras or inertial sensors to avoid an intermediary device, but they tend to perform well only for unobscured or overt movements. Brain computer interfaces (BCIs) have been imagined for decades to solve the interface problem by allowing for input to computers via thought alone. However high-bandwidth communication has only been demonstrated using invasive BCIs with decoders designed for single individuals, and so cannot scale to the general public. In contrast, neuromotor signals found at the muscle offer access to subtle gestures and force information. Here we describe the development of a noninvasive neuromotor interface that allows for computer input using surface electromyography (sEMG). We developed a highly-sensitive and robust hardware platform that is easily donned/doffed to sense myoelectric activity at the wrist and transform intentional neuromotor commands into computer input. We paired this device with an infrastructure optimized to collect training data from thousands of consenting participants, which allowed us to develop generic sEMG neural network decoding models that work across many people without the need for per-person calibration. Test users not included in the training set demonstrate closed-loop median performance of gesture decoding at 0.5 target acquisitions per second in a continuous navigation task, 0.9 gesture detections per second in a discrete gesture task, and handwriting at 17.0 adjusted words per minute. We demonstrate that input bandwidth can be further improved up to 30% by personalizing sEMG decoding models to the individual, anticipating a future in which humans and machines co-adapt to provide seamless translation of human intent. To our knowledge this is the first high-bandwidth neuromotor interface that directly leverages biosignals with performant out-of-the-box generalization across people.

Making gestural input from arm-worn inertial sensors more practical

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

Gesture Recognition with a 3-D Accelerometer

Gesture recognition by instantaneous surface EMG images

Finger Angle-Based Hand Gesture Recognition for Smart Infrastructure Using Wearable Wrist-Worn Camera

Designing and Evaluating Hand-to-Hand Gestures with Dual Commodity Wrist-Worn Devices

Wireless Hand Gesture Recognition Based on Continuous-Wave Doppler Radar Sensors

Generalizable gesture recognition using magnetomyography

Acoustic Sensing-based Hand Gesture Detection for Wearable Device Interaction

Gesture Recognition Framework for Teleoperation of Infrared (IR) Consumer Devices Using a Novel pFMG Soft Armband

Design and evaluation of a hand gesture recognition approach for real-time interactions

Cyber–Physical Mobile Arm Gesture Recognition using Ultrasound and Motion Data

Hand Gesture Recognition by a MMG-Based Wearable Device

Hand Gesture Recognition on a Resource-Limited Interactive Wristband

Real-Time Gesture Recognition with Virtual Glove Markers

Anatomically Designed Triboelectric Wristbands with Adaptive Accelerated Learning for Human–Machine Interfaces

A generic noninvasive neuromotor interface for human-computer interaction

Implementation of Hand Gesture Recognition Device Applicable to Smart Watch Based on Flexible Epidermal Tactile Sensor Array

Adaptive and Personalized Gesture Recognition Using Textile Capacitive Sensor Arrays

Ultigesture: A Wristband-based Platform for Continuous Gesture Control in Healthcare

A Novel Data Glove Using Inertial and Magnetic Sensors for Motion Capture and Robotic Arm-Hand Teleoperation