Abstract:Purpose: Mild to profound hearing impairment places limits on effective communication and day-to-day interaction. Sign language, being the primary mode of communication for people with hearing loss, lacks communicative efficacy. A wearable assistive device that aims to convert sign language into speech is proposed to facilitate communication between the unimpaired population (untrained in sign language) and the hearing impaired population. However, the wide use of geo-centric sign languages in India has resulted in the lack of standardised sign-language datasets. In the proposed work, a compact, low-resource, motion sensor-based, wireless, single and double hand-gesture recognition module is designed to address this issue. Material and methods: The proposed module is designed to perform a two-step process with a Hidden Markov Model (HMM) based gesture-to-text conversion and a bilingual text-to-speech synthesis. Multi-threading based parallel processing is implemented to enable simultaneous working of the two systems to reduce the delay. In the proposed continuous-gesture recognition system, non-gesture hand motions are modelled using ergodic HMMs that are trained by concatenating all the states of gesture models, allowing equiprobable transitions. The proposed system is modelled and tested for American Sign Language (ASL) and user-defined gestures. Results: The maximum performance of the proposed system in recognising single-handed and double-handed gestures in terms of F1-score is 98.17% and 84.85%, respectively. Further, the proposed system achieves a maximum F1-score of 98% and 83% in recognising isolated and continuous gestures, respectively. Conclusion: The gesture-to-speech conversion system is ported on Raspberry Pi making the proposed system wireless, and highly mobile.Implications for rehabilitationThe research work proposes to develop a gesture-to-speech conversion system to enable the deaf-mute population in communication. The major implications of the proposed work are:•A light-weight raspberry pi 3B + module hosts the entire hardware andsoftware, and is sufficient to train and test the gesture-to-speech conversion system, thereby ensuring greater mobility.•The proposed system can be customised to recognise user-defined gestures with just 5 examples of the new gesture.•The proposed system can be expanded to control home appliances (IoT applications) by combining the output of the proposed gesture recognition system with appropriate control interfaces.

American Sign Language Translation via Smart Wearable Glove Technology

Smart Glove for Sign Language Translation

A new architecture for hand-worn Sign language to Speech translator

Intersign Glove: A Communication Interface for the Aid of Deaf and Mute Community

Dataglove for Sign Language Recognition of People with Hearing and Speech Impairment via Wearable Inertial Sensors

American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

Intelligent gloves: An IT intervention for deaf-mute people

SignSpeak: Open-Source Time Series Classification for ASL Translation

AI-Based Sensory Glove System to Recognize Bengali Sign Language (BaSL)

Sign Language to Text Conversion

Soft Sign Language Interpreter on Your Skin

Development of a low-resource wearable continuous gesture-to-speech conversion system

SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator

Sign Language Conversation Interpretation Using Wearable Sensors and Machine Learning

Bridging Communication Gaps: A Comprehensive Survey on Sign Language Hand Gesture Smart Recognition Technology

GLOS: GLOve for Speech Recognition

Novel Wearable System to Recognize Sign Language in Real Time

Smart Hand Gloves for Deaf and Dumb People

Multimodal Hand/finger Movement Sensing and Fuzzy Encoding for Data‐efficient Universal Sign Language Recognition

Sign Language Recognition System using Flex Sensor Network

American Sign Language Recognition Using RF Sensing