Abstract:Purpose: The article presents a design and development of a generic assistive system to establish an independent conversation-platform for hearing-speech impaired and visually impaired persons. Materials: The developed software system is accomplished through programming using python and html. Methods: Considering the constraints associated to the above mentioned impairments, the system implements both speech-to-text/gesture and text/gesture-to-speech conversion in its operation. In real-time hand-gesture to speech generation process is implemented using static image tracking, CNN based deep learning technique and MediaPipe hand-tracking solution. The software-prototype-terminals can be accessed through internet using MQTT protocol to accomplish the communicative conversation between visually impaired and hearing-speech impaired persons. Results: The software system exhibits an average prediction time of less than approximately 1 s and 2 s for a four-letter based audio-word and a single hand-gesture, respectively, which are commensurate to the average time-complexity during human-to-human conversation. The average accuracy and loss for the hand-gestures through the CNN based deep learning are 0.9996 and 0.0008, respectively. The confusion matrix related to the prediction of alphabet-specific hand-gestures shows its satisfactory performance in gesture recognition. Conclusions: The software-prototype of the generic assistive device shows its potential to establish an exclusive communication between a visually impaired and a hearing-speech impaired person through the internet. The same software-interface can also be used to accomplish a communicative conversation between either only visually-impaired persons or only hearing-speech impaired persons. IMPLICATIONS FOR REHABILITATIONThe article presents a design and development of a generic assistive interface to establish an independent conversation-platform for hearing-speech impaired and visually impaired people via internet network.The same software-interface can also be used to accomplish a communicative conversation between either only visually-impaired persons or only hearing-speech impaired persons.The design can be further extended by incorporating multi-modal impairments to make a universal assistive device for all-in-one communication.

Design of Intelligent Human-Computer Interaction System for Hard of Hearing and Non-Disabled People

Right mix of speech and non-speech: hybrid auditory feedback in mobility assistance of the visually impaired

Design of voice interaction-based training system for cerebral palsy rehabilitation

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

Design of a speech interaction system for man-machine confrontation

A Portable Wireless Eye Movement-Controlled Human-Computer Interface for the Disabled

Building Language Communication between Deaf People and Hearing Society through Multimodal Human-Computer Interface

An assistive interface protocol for communication between visually and hearing-speech impaired persons in internet platform

Application of User Experience and Design Thinking to the Construction of a Class Assistance System for Hearing- and Speech-Impaired People

Handtalker: A Multimodal Dialog System Using Sign Language and 3-D Virtual Human

Intelligent gloves: An IT intervention for deaf-mute people

HCI Design for People with Visual Disability in Social Interaction.

Improving communication and promoting social inclusion for hearing-impaired users: Usability evaluation and design recommendations for assistive mobile applications

Development of A Human Computer Interaction System Based on Multi-Modal Gaze Tracking Methods

A Bi-directional Interactive System of Sign Language and Visual Speech Based on Portable Devices.

Design and Implementation of Speech Interaction System in Pervasive Computing Environment

Human-Computer Chinese Sign Language Interaction System

A Sign Language Interaction System Based on Pneumatic Soft Hand

A Wearable Vision-To-Audio Sensory Substitution System Based on Deep Learning for the Visually Impaired

Construction Of A Voice Driven Life Assistant System For Visually Impaired People

Decoding Silent Speech Commands from Articulatory Movements Through Soft Magnetic Skin and Machine Learning