Abstract:Purpose: Mild to profound hearing impairment places limits on effective communication and day-to-day interaction. Sign language, being the primary mode of communication for people with hearing loss, lacks communicative efficacy. A wearable assistive device that aims to convert sign language into speech is proposed to facilitate communication between the unimpaired population (untrained in sign language) and the hearing impaired population. However, the wide use of geo-centric sign languages in India has resulted in the lack of standardised sign-language datasets. In the proposed work, a compact, low-resource, motion sensor-based, wireless, single and double hand-gesture recognition module is designed to address this issue. Material and methods: The proposed module is designed to perform a two-step process with a Hidden Markov Model (HMM) based gesture-to-text conversion and a bilingual text-to-speech synthesis. Multi-threading based parallel processing is implemented to enable simultaneous working of the two systems to reduce the delay. In the proposed continuous-gesture recognition system, non-gesture hand motions are modelled using ergodic HMMs that are trained by concatenating all the states of gesture models, allowing equiprobable transitions. The proposed system is modelled and tested for American Sign Language (ASL) and user-defined gestures. Results: The maximum performance of the proposed system in recognising single-handed and double-handed gestures in terms of F1-score is 98.17% and 84.85%, respectively. Further, the proposed system achieves a maximum F1-score of 98% and 83% in recognising isolated and continuous gestures, respectively. Conclusion: The gesture-to-speech conversion system is ported on Raspberry Pi making the proposed system wireless, and highly mobile.Implications for rehabilitationThe research work proposes to develop a gesture-to-speech conversion system to enable the deaf-mute population in communication. The major implications of the proposed work are:•A light-weight raspberry pi 3B + module hosts the entire hardware andsoftware, and is sufficient to train and test the gesture-to-speech conversion system, thereby ensuring greater mobility.•The proposed system can be customised to recognise user-defined gestures with just 5 examples of the new gesture.•The proposed system can be expanded to control home appliances (IoT applications) by combining the output of the proposed gesture recognition system with appropriate control interfaces.

Intersign Glove: A Communication Interface for the Aid of Deaf and Mute Community

Intelligent gloves: An IT intervention for deaf-mute people

A new architecture for hand-worn Sign language to Speech translator

Smart Glove for Sign Language Translation

American Sign Language Translation via Smart Wearable Glove Technology

Dataglove for Sign Language Recognition of People with Hearing and Speech Impairment via Wearable Inertial Sensors

Innovative Device for Enhancing Deaf-Mute Persons Communication Possibilities

Smart Hand Gloves for Deaf and Dumb People

GLOS: GLOve for Speech Recognition

Artificial Intelligence-Enabled Gesture-Language-Recognition Feedback System Using Strain-Sensor-Arrays-Based Smart Glove.

Artificial Intelligence Biosensing System on Hand Gesture Recognition for the Hearing Impaired

Bridging Communication Gaps: A Comprehensive Survey on Sign Language Hand Gesture Smart Recognition Technology

Development of a low-resource wearable continuous gesture-to-speech conversion system

A Simple Sign Language Recognition System Based on Data Glove

Talking Hands: Speech to Sign Language Converter

AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

HAND SIGN LANGUAGE RECOGNITION USING AUGMENTED REALITY & MACHINE LEARNING

An ARM-based embedded gesture recognition system using a data glove

Soft Sign Language Interpreter on Your Skin

Sign Language to Text Conversion

Intelligent Gloves for Assisting Individuals with Visual Impairment