Abstract:Initiative service is a key research direction for the new generation of service robots. It is important to automatically track humans for initiative service in human-robot interaction. To solve the problems of low precision and poor anti-interference capability of only using single-modal (audio or visual) information, a speaker positioning and tracking method based on an audio-visual bimodal combination is proposed. First, the azimuth of the speaker is obtained based on the time difference of arrival using a microphone array, and face detection based on AdaBoost is carried out using the camera. A distance and azimuth calculation model is established to obtain the position of the speaker. Second, a speaker positioning strategy based on an audio-visual bimodal combination is designed to handle different situations. Third, the path is planned by which the azimuth and distance between the robot and the speaker are maintained in a limited range. Different azimuths and distances for speaker tracking are set to perform various simulations. Finally, the mobile robot is driven to follow the path using the STM32 real-time control system. Information from the microphone array and the camera is collected and processed by Raspberry Pi. The tracking accuracy was tested under a single-face situation by setting 20 different target points, and 10 tests were carried out under each point. Under multi-face situations, the audio-visual bimodal information is combined to identify the speaker, and then the Kalman filter is used in face tracking. The experimental results demonstrate that the running trajectory of the mobile robot is close to the ideal trajectory, which ensures effective speaker tracking.

Detect The Possible Spokesperson With An Omni-Directional Camera, In A Robot-Human Communication System

Human Body Recognition And Positioning With Multiple Cameras Based On "Vibration Signals" From Skin Surfaces

Hand Posture Recognition and Tracking Based on Bag-of-Words for Human Robot Interaction

Gesture recognition using combination of acceleration sensor and images for casual communication between robots and humans

Audio-Visual Bimodal Combination-Based Speaker Tracking Method for Mobile Robot

Omni-directional Vision Based Human Motion Detection for Autonomous Mobile Robots

Human–robot non-verbal interaction empowered by real-time auditory and visual multiple-talker tracking

Omnidirectional Vision for Mobile Robot Human Body Detection and Localization

Multimodal Gesture Recognition for Mascot Robot System Based on Choquet Integral Using Camera and 3D Accelerometers Fusion

Selfie Taking with Facial Expression Recognition Using Omni-directional Camera

Visual Perception for Multiple Human–Robot Interaction From Motion Behavior

A Flexible Framework for Virtual Omnidirectional Vision to Improve Operator Situation Awareness

Vision-Guided Robot Hearing

Intelligent communication of two humanoid robots based on computer vison

Multimodal Approach to Human-Face Detection and Tracking

Vision-aided Speech Recognition System for a Small Four-Legged Robot

Environment Recognition Using Robot Camera

Design and implementation of an omnidirectional vision system for robot perception

Combined 2-D sound source localization with stereo vision for intelligent Human-Robot Interaction of service robot

Motion Detection for Human Bodies Basing Adaptive Background Subtraction by Using an Omnidirectional Camera

Amplifying robotics capacities with a human touch: An immersive low-latency panoramic remote system