Abstract:With the global spread of the novel coronavirus, avoiding human-to-human contact has become an effective way to cut off the spread of the virus. Therefore, contactless gesture recognition becomes an effective means to reduce the risk of contact infection in outbreak prevention and control. However, the recognition of everyday behavioral sign language of a certain population of deaf people presents a challenge to sensing technology. Ubiquitous acoustics offer new ideas on how to perceive everyday behavior. The advantages of a low sampling rate, slow propagation speed, and easy access to the equipment have led to the widespread use of acoustic signal-based gesture recognition sensing technology. Therefore, this paper proposed a contactless gesture and sign language behavior sensing method based on ultrasonic signals-UltrasonicGS. The method used Generative Adversarial Network (GAN)-based data augmentation techniques to expand the dataset without human intervention and improve the performance of the behavior recognition model. In addition, to solve the problem of inconsistent length and difficult alignment of input and output sequences of continuous gestures and sign language gestures, we added the Connectionist Temporal Classification (CTC) algorithm after the CRNN network. Additionally, the architecture can achieve better recognition of sign language behaviors of certain people, filling the gap of acoustic-based perception of Chinese sign language. We have conducted extensive experiments and evaluations of UltrasonicGS in a variety of real scenarios. The experimental results showed that UltrasonicGS achieved a combined recognition rate of 98.8% for 15 single gestures and an average correct recognition rate of 92.4% and 86.3% for six sets of continuous gestures and sign language gestures, respectively. As a result, our proposed method provided a low-cost and highly robust solution for avoiding human-to-human contact.

SonicOperator: Ultrasonic gesture recognition with deep neural network on mobiles

Hand gesture recognition scheme based on millimeter-wave radar with convolutional neural network

Poster - Sonicnect: Accurate Hands-Free Gesture Input System With Smart Acoustic Sensing

DopGest: Dual-Frequency Based Ultrasonic Gesture Recognition

Pervasive Hand Gesture Recognition for Smartphones using Non-audible Sound and Deep Learning

Interacting with Soli

UltrasonicGS: A Highly Robust Gesture and Sign Language Recognition Method Based on Ultrasonic Signals

UltraGesture: Fine-Grained Gesture Sensing and Recognition

Hand Gesture Recognition for Smart Devices by Classifying Deterministic Doppler Signals

Micro Hand Gesture Recognition System Using Ultrasonic Active Sensing

SignID: Acoustic-based Identification with Single Sign Gesture

HCI on the Table: Robust Gesture Recognition Using Acoustic Sensing in Your Hand

SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks

Doppler-Radar Based Hand Gesture Recognition System Using Convolutional Neural Networks

Gesture Recognition Using Visible Light on Mobile Devices

Angle-of-Arrival-Based Gesture Recognition Using Ultrasonic Multi-Frequency Signals

Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic Signals

RadarNet: Efficient Gesture Recognition Technique Utilizing a Miniature Radar Sensor

Ultrasound-Based 3-D Gesture Recognition: Signal Optimization, Trajectory, and Feature Classification

Efficient High Cross-User Recognition Rate Ultrasonic Hand Gesture Recognition System

Dynamic Gesture Recognition Based on RF Sensor and AE-LSTM Neural Network