Abstract:This paper designed a voice interactive robot system that can conveniently execute assigned service tasks in real-life scenarios. It is equipped without a microphone where users can control the robot with spoken commands; the voice commands are then recognized by a well-trained deep neural network model of automatic speech recognition (ASR), which enables the robot to execute and complete the command based on the navigation of a real-time simultaneous localization and mapping (SLAM) algorithm. The voice interaction recognition model is divided into two parts: (1) speaker separation and (2) ASR. The speaker separation is applied by a deep-learning system consisting of eight convolution layers, one LSTM layer, and two fully connected (FC) layers to separate the speaker’s voice. This model recognizes the speaker’s voice as a referrer that separates and holds the required voiceprint and removes noises from other people’s voiceprints. Its automatic speech recognition uses the novel sandwich-type conformer model with a stack of three layers, and combines convolution and self-attention to capture short-term and long-term interactions. Specifically, it contains a multi-head self-attention module to directly convert the voice data into text for command realization. The RGB-D vision-based camera uses a real-time appearance-based mapping algorithm to create the environment map and replace the localization with a visional odometer to allow the robot to navigate itself. Finally, the proposed ASR model was tested to check if the desired results will be obtained. Performance analysis was applied to determine the robot’s environment isolation and voice recognition abilities. The results showed that the practical robot system successfully completed the interactive service tasks in a real environment. This experiment demonstrates the outstanding performance with other ASR methods and voice control mobile robot systems. It also verified that the designed voice interaction recognition system enables the mobile robot to execute tasks in real-time, showing that it is a convenient way to complete the assigned service applications.

Design of the robotic system for human-robot interaction using sound source localization, mapping data and voice recognition

Combined 2-D sound source localization with stereo vision for intelligent Human-Robot Interaction of service robot

Design of Kinect-Based Human Robot Interaction Systems for A Robocup Middle Size League Soccer Robot

Design of a speech interaction system for man-machine confrontation

Design and Implementation of the Voice Command Recognition and the Sound Source Localization System for Human–Robot Interaction

The Design and Implementation of Natural Human-Robot Interaction System Based on Kinect Sensor

Sound-Source Tracking and Obstacle Avoidance System for the Mobile Robot

Voice Interaction Recognition Design in Real-Life Scenario Mobile Robot Applications

Audio-Visual Bimodal Combination-Based Speaker Tracking Method for Mobile Robot

Contactless Interaction System Based on Facial Expression Recognition for Humanoid Piano Robot

Audio–Visual Sound Source Localization and Tracking Based on Mobile Robot for The Cocktail Party Problem

Remote Control System of Spherical Robot Based on Silent Speech Recognition

Human–robot non-verbal interaction empowered by real-time auditory and visual multiple-talker tracking

Design of intelligent robot control system based on human–computer interaction

Human-assisted Sound Event Recognition for Home Service Robots.

Auditory System for a Mobile Robot

Human–robot interaction via voice-controllable intelligent user interface

Sound source localization for auditory perception of a humanoid robot using deep neural networks

Implementation of Engagement Detection for Human–Robot Interaction in Complex Environments

Effects of Robot Sound on Auditory Localization in Human-Robot Collaboration

Practical Robotic Auditory Perception and Approaching Methods Based on Small-sized Microphone Array