Abstract:As the number of electronic gadgets in our daily lives is increasing and most of them require some kind of human interaction, this demands innovative, convenient input methods. There are limitations to state-of-the-art (SotA) ultrasound-based hand gesture recognition (HGR) systems in terms of robustness and accuracy. This research presents a novel machine learning (ML)-based end-to-end solution for hand gesture recognition with low-cost micro-electromechanical (MEMS) system ultrasonic transducers. In contrast to prior methods, our ML model processes the raw echo samples directly instead of using pre-processed data. Consequently, the processing flow presented in this work leaves it to the ML model to extract the important information from the echo data. The success of this approach is demonstrated as follows. Four MEMS ultrasonic transducers are placed in three different geometrical arrangements. For each arrangement, different types of ML models are optimized and benchmarked on datasets acquired with the presented custom hardware (HW): convolutional neural networks (CNNs), gated recurrent units (GRUs), long short-term memory (LSTM), vision transformer (ViT), and cross-attention multi-scale vision transformer (CrossViT). The three last-mentioned ML models reached more than 88% accuracy. The most important innovation described in this research paper is that we were able to demonstrate that little pre-processing is necessary to obtain high accuracy in ultrasonic HGR for several arrangements of cost-effective and low-power MEMS ultrasonic transducer arrays. Even the computationally intensive Fourier transform can be omitted. The presented approach is further compared to HGR systems using other sensor types such as vision, WiFi, radar, and state-of-the-art ultrasound-based HGR systems. Direct processing of the sensor signals by a compact model makes ultrasonic hand gesture recognition a true low-cost and power-efficient input method.

Micro Hand Gesture Recognition System Using Ultrasonic Active Sensing

Dynamic hand gesture recognition using hidden Markov models

Hand gesture recognition scheme based on millimeter-wave radar with convolutional neural network

Efficient High Cross-User Recognition Rate Ultrasonic Hand Gesture Recognition System

Wireless Hand Gesture Recognition Based on Continuous-Wave Doppler Radar Sensors

End-to-End Ultrasonic Hand Gesture Recognition

Hand Gesture Recognition Using Ultrasonic Array with Machine Learning

Mixed Hand Gesture Recognition System And Its Application

Hand Gesture Recognition Using Ultrasonic Waves

M-Gesture : Person-Independent Real-Time In-Air Gesture Recognition Using Commodity Millimeter Wave Radar

Dynamic Hand Gesture Classification Based On Radar Micro-Doppler Signatures

Hand Gesture Recognition for Smart Devices by Classifying Deterministic Doppler Signals

Design of hand gesture recognition system for human-computer interaction

Dynamic Hand Gesture Recognition Based on Micro-Doppler Radar Signatures Using Hidden Gauss–Markov Models

A Real Time Vision-Based Hand Gestures Recognition System

3D Intuitive Gesture Interaction via Motion Sensing

Sensing, Tracking, and Recognition of Macro–Micro Hand Gestures Using Interferometric MIMO Radar

Lightweight Online Semisupervised Learning for Ultrasonic Radar-Based Dynamic Hand Gesture Recognition

UltraGesture: Fine-Grained Gesture Sensing and Recognition

SignID: Acoustic-based Identification with Single Sign Gesture

Long-Range Gesture Recognition Using Millimeter Wave Radar