Abstract:Human-robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people's lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

Emotional Communication Robot Based on 3D Face Model and ASR Technology

Design and Realization of a Multi-DoF Robotic Head for Affective Humanoid Facial Expression Imitation.

Emotional States Based 3-D Fuzzy Atmosfield for Casual Communication Between Humans and Robots

A Facial Expression Emotion Recognition Based Human-robot Interaction System

A Multimodal Emotional Communication Based Humans-Robots Interaction System

Contactless Interaction System Based on Facial Expression Recognition for Humanoid Piano Robot

ExpressionBot: An Emotive Lifelike Robotic Face for Face-to-Face Communication

Multi-Modal Based Fuzzy Atmosfield in Human-Robot Interaction

Communication Atmosphere in Humans and Robots Interaction Based on Fuzzy Analytical Hierarchy Process

Dynamic Emotion Understanding in Human–Robot Interaction Based on Two-Layer Fuzzy SVR-TS Model

Human-Robot Emotional Interaction Model Based on Reinforcement Learning

Improving Human-Robot Interaction by Enhancing NAO Robot Awareness of Human Facial Expression

A Fast and Accurate Multi-Model Facial Expression Recognition Method for Affective Intelligent Robots

Research of the Robot Head System with Facial Expression Recognition and Representation

Human-robot facial coexpression

Intelligent Facial Emotion Recognition and Semantic-Based Topic Detection for A Humanoid Robot

A Lightweight Network-Based Sign Language Robot with Facial Mirroring and Speech System

Automatic Generation of Robot Facial Expressions with Preferences.

EmoFace: Audio-driven Emotional 3D Face Animation

An Automated Research for Emotion Recognition and Generation

Emotional Chinese talking head system