Abstract:The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.

Robust Real-Time Human Perception with Depth Camera.

A Two-Stage Human Body Detector on Depth Data

Video Based Head Detection and Tracking Surveillance System

An Intelligent Surveillance System Base on Accelerated Human and Face Detection

Unconstrained Face Detection and Recognition Based on Rgb-D Camera for the Visually Impaired

Real-time multiple human perception with color-depth cameras on a mobile robot

Robust 3-D Human Detection in Complex Environments With a Depth Camera.

An ultra-fast human detection method for color-depth camera

Reliably Detecting Humans in Crowded and Dynamic Environments Using RGB-D Camera

Detecting and tracking people in real time with RGB-D camera

Real-time human detection and tracking in complex environments using single RGBD camera

Real-Time Monocular Human Depth Estimation and Segmentation on Embedded Systems

Robust Human Detection with Super-Pixel Segmentation and Random Ferns Classification Using RGB-D Camera

Fast Human Detection in RGB-D Images Based on Color-Depth Joint Feature Learning.

Real-time Monocular 3D People Localization and Tracking on Embedded System

Robust Bilayer Segmentation and Motion/Depth Estimation with a Handheld Camera

Human detection using depth information

Face-from-Depth for Head Pose Estimation on Depth Images

Depth-aware gaze-following via auxiliary networks for robotics

A Top-View Multiple People Tracking System Based on Newest YOLOv5 and DeepSort Using Depth Data

Physical Blob Detector and Multi-Channel Color Shape Descriptor for Human Detection.