Abstract:The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.

Reliably Detecting Humans in Crowded and Dynamic Environments Using RGB-D Camera

Detecting and tracking people in real time with RGB-D camera

Reliably Detecting Humans With Rgb-D Camera With Physical Blob Detector Followed By Learning-Based Filtering

Robust Human Detection with Super-Pixel Segmentation and Random Ferns Classification Using RGB-D Camera

Real-time human detection and tracking in complex environments using single RGBD camera

Fast Human Detection in RGB-D Images Based on Color-Depth Joint Feature Learning.

A Two-Stage Human Body Detector on Depth Data

Real-time Detection and Tracking Algorithm Based on the Color and Character of Human Face

A Novel Method For Real-Time Object Detection And Multiple Persons Tracking

Multi-objects Real Time Recognition Based on Color Information

3D-SSD: Learning Hierarchical Features from RGB-D Images for Amodal 3D Object Detection

An ultra-fast human detection method for color-depth camera

Robust Real-Time Human Perception with Depth Camera.

Robust 3-D Human Detection in Complex Environments With a Depth Camera.

Physical Blob Detector and Multi-Channel Color Shape Descriptor for Human Detection.

Dynamic human object recognition by combining color and depth information with a clothing image histogram

Detecting Humans in RGB-D Data with CNNs

Multiple People Extraction Using 3D Range Sensor

RGB-D Human Identification and Tracking in a Smart Environment

Real-time multiple human perception with color-depth cameras on a mobile robot

Feature Regions Segmentation Based RGB-D Visual Odometry in Dynamic Environment