Abstract:Many technologies for human-computer interaction have been designed for hearing individuals and depend upon vocalized speech, precluding users of American Sign Language (ASL) in the Deaf community from benefiting from these advancements. While great strides have been made in ASL recognition with video or wearable gloves, the use of video in homes has raised privacy concerns, while wearable gloves severely restrict movement and infringe on daily life. Methods: This paper proposes the use of RF sensors for HCI applications serving the Deaf community. A multi-frequency RF sensor network is used to acquire non-invasive, non-contact measurements of ASL signing irrespective of lighting conditions. The unique patterns of motion present in the RF data due to the micro-Doppler effect are revealed using time-frequency analysis with the Short-Time Fourier Transform. Linguistic properties of RF ASL data are investigated using machine learning (ML). Results: The information content, measured by fractal complexity, of ASL signing is shown to be greater than that of other upper body activities encountered in daily living. This can be used to differentiate daily activities from signing, while features from RF data show that imitation signing by non-signers is 99\% differentiable from native ASL signing. Feature-level fusion of RF sensor network data is used to achieve 72.5\% accuracy in classification of 20 native ASL signs. Implications: RF sensing can be used to study dynamic linguistic properties of ASL and design Deaf-centric smart environments for non-invasive, remote recognition of ASL. ML algorithms should be benchmarked on native, not imitation, ASL data.

Attention-Augmented Electromagnetic Representation of Sign Language for Human-Computer Interaction in Deaf-and-Mute Community

Applying Mmwave Radar Sensors to Vocabulary-Level Dynamic Chinese Sign Language Recognition for the Community with Deafness and Hearing Loss

EM-Sign: A Non-Contact Recognition Method Based on 24 GHz Doppler Radar for Continuous Signs and Dialogues

Sentence-Level Sign Language Recognition Using RF signals

American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

American Sign Language Recognition Using RF Sensing

Automatic recognition of sign language subwords based on portable accelerometer and EMG sensors

A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation

Overview of Electromagnetic Representation and Interpretation of Sign Language Semantics

Real-Time Vision-Based Chinese Sign Language Recognition with Pose Estimation and Attention Network

Sign language translation system based on micro-inertial measurement units and ZigBee network

Sign Language Recognition Based On Hmm/Ann/Dp

A two-way translation system of Chinese sign language based on computer vision

SignFind: A Synchronized Sign Language and Chinese Character Teaching Game for Chinese Deaf Children Using Gesture Recognition

Building Language Communication between Deaf People and Hearing Society through Multimodal Human-Computer Interface

Improvement of Chinese sign language translation system based on collaboration of arm and finger sensing nodes

Expressive ASL Recognition using Millimeter-wave Wireless Signals

DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

MyoSign

Chinese Sign Language Recognition Based on DTW-Distance-Mapping Features

A Method for Sign Language Recognition Based on Information Fusion of Accelerometer and Surface Electromyography Sensors