Abstract:The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.

Efficient Face Detection with Audio-Based Region Proposals for Human-Robot Interactions

An Intelligent Surveillance System Base on Accelerated Human and Face Detection

Addressee Detection Using Facial and Audio Features in Mixed Human–Human and Human–Robot Settings: A Deep Learning Framework

Vision-Guided Robot Hearing

Real-time Architecture for Audio-Visual Active Speaker Detection.

Audio-Visual Bimodal Combination-Based Speaker Tracking Method for Mobile Robot

Multimodal Approach to Human-Face Detection and Tracking

Implementing a Face Detection System Practically Robust against Size Variation and Brightness Fluctuation for Distributed Autonomous Human Supporting Robotic Environment

On effective human robot interaction based on recognition and association

Vision- and tactile-based continuous multimodal intention and attention recognition for safer physical human-robot interaction

Research on Sound Source Localization and Real-Time Facial Expression Recognition for Security Robot

A Real-Time Face Detection and Recognition System for a Mobile Robot in a Complex Background

A Real Time Face Detection Method in Human-Machine Interaction

MOS: A Low Latency and Lightweight Framework for Face Detection, Landmark Localization, and Head Pose Estimation

Real-time multiple human perception with color-depth cameras on a mobile robot

Human–robot non-verbal interaction empowered by real-time auditory and visual multiple-talker tracking

Cross-Modal Analysis of Human Detection for Robotics: An Industrial Case Study

Attention-Oriented Action Recognition for Real-Time Human-Robot Interaction

Visual Perception for Multiple Human–Robot Interaction From Motion Behavior

A Near-Real-Time Processing Ego Speech Filtering Pipeline Designed for Speech Interruption During Human-Robot Interaction

Real-Time Human Intrusion Detection Using Audio-Visual Fusion.