Abstract:Human-robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people's lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator

Hear Sign Language: A Real-Time End-to-End Sign Language Recognition System

Demo: The Sound of Silence: End-to-End Sign Language Recognition Using SmartWatch.

WearSign: Pushing the Limit of Sign Language Translation Using Inertial and EMG Wearables.

SmartASL

TransASL: A Smart Glass Based Comprehensive ASL Recognizer in Daily Life.

HearASL: Your Smartphone Can Hear American Sign Language

American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

SonicASL

Sign language translation system based on micro-inertial measurement units and ZigBee network

A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation

A Bi-directional Interactive System of Sign Language and Visual Speech Based on Portable Devices.

A wearable system for sign language recognition enabled by a convolutional neural network

SignFind: A Synchronized Sign Language and Chinese Character Teaching Game for Chinese Deaf Children Using Gesture Recognition

SignSpeak: Open-Source Time Series Classification for ASL Translation

L-Sign: Large-Vocabulary Sign Gestures Recognition System

End-to-End Two-Handed Sign Language Translation

MyoSign: enabling end-to-end sign language recognition with wearables

Automatic recognition of sign language subwords based on portable accelerometer and EMG sensors

A Lightweight Network-Based Sign Language Robot with Facial Mirroring and Speech System

Sign Language Conversation Interpretation Using Wearable Sensors and Machine Learning