Abstract:Purpose: The article presents a design and development of a generic assistive system to establish an independent conversation-platform for hearing-speech impaired and visually impaired persons. Materials: The developed software system is accomplished through programming using python and html. Methods: Considering the constraints associated to the above mentioned impairments, the system implements both speech-to-text/gesture and text/gesture-to-speech conversion in its operation. In real-time hand-gesture to speech generation process is implemented using static image tracking, CNN based deep learning technique and MediaPipe hand-tracking solution. The software-prototype-terminals can be accessed through internet using MQTT protocol to accomplish the communicative conversation between visually impaired and hearing-speech impaired persons. Results: The software system exhibits an average prediction time of less than approximately 1 s and 2 s for a four-letter based audio-word and a single hand-gesture, respectively, which are commensurate to the average time-complexity during human-to-human conversation. The average accuracy and loss for the hand-gestures through the CNN based deep learning are 0.9996 and 0.0008, respectively. The confusion matrix related to the prediction of alphabet-specific hand-gestures shows its satisfactory performance in gesture recognition. Conclusions: The software-prototype of the generic assistive device shows its potential to establish an exclusive communication between a visually impaired and a hearing-speech impaired person through the internet. The same software-interface can also be used to accomplish a communicative conversation between either only visually-impaired persons or only hearing-speech impaired persons. IMPLICATIONS FOR REHABILITATIONThe article presents a design and development of a generic assistive interface to establish an independent conversation-platform for hearing-speech impaired and visually impaired people via internet network.The same software-interface can also be used to accomplish a communicative conversation between either only visually-impaired persons or only hearing-speech impaired persons.The design can be further extended by incorporating multi-modal impairments to make a universal assistive device for all-in-one communication.

AidServer: Design and Research of a Communication Accessibility Service System for Hearing Impaired Servers in Silent Restaurants

Right mix of speech and non-speech: hybrid auditory feedback in mobility assistance of the visually impaired

Hybrid auditory feedback: a new method for mobility assistance of the visually impaired.

Silent Delivery: Make Instant Delivery More Accessible for the DHH Delivery Workers Through Sensory Substitution.

A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation

Using web services to realize remote hearing assessment

Application of User Experience and Design Thinking to the Construction of a Class Assistance System for Hearing- and Speech-Impaired People

Technology Challenges and Aids: The Sustainable Development of Professional Interpreters in Listening Comprehension Effectiveness and Interpreting Performance

Design of Intelligent Human-Computer Interaction System for Hard of Hearing and Non-Disabled People

Business Model Oriented Towards Solutions to Improve the Communication Efficiency of Deaf-Mute People

A Multimodal Assistive System for Helping Visually Impaired in Social Interactions

Lightweight smart glass system with audio aid for visually impaired people

A Study on Communication Platforms and Services Needed and Available for Mute and Deaf Community

Deaf-aid

Skin-Integrated Wireless Odor Message Delivery Electronics for the Deaf-blind

Improving communication and promoting social inclusion for hearing-impaired users: Usability evaluation and design recommendations for assistive mobile applications

An assistive interface protocol for communication between visually and hearing-speech impaired persons in internet platform

Mobile Sound Recognition for the Deaf and Hard of Hearing

Designing AACs for People with Aphasia Dining in Restaurants

Sign Language e-Learning system for hearing-impaired community of Pakistan

Silent no more: a comprehensive review of artificial intelligence, deep learning, and machine learning in facilitating deaf and mute communication