Abstract:The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.

Efficient Hierarchical Markov Random Fields for Object Detection on a Mobile Robot

Robust Object Tracking with a Hierarchical Ensemble Framework

Multi-view Aggregation for Real-Time Accurate Object Detection of a Moving Camera

A Real-Time Face Detection and Recognition System for a Mobile Robot in a Complex Background

Detecting More Objects Indoors: a Depth-guided Detector for Mobile Robots

Rapid Detection of Moving Target in the Competitive Networked Robots

A Novel Video Object Tracking Approach Based on Kernel Density Estimation and Markov Random Field

Unknown Object Detection Using a One-Class Support Vector Machine for a Cloud-Robot System

VMM: Viewpoint-based Memory Mechanism for Object Detection of Moving Sensors

MobileNet-JDE: a lightweight multi-object tracking model for embedded systems

Real-time multiple human perception with color-depth cameras on a mobile robot

CNN-based Omnidirectional Object Detection for HermesBot Autonomous Delivery Robot with Preliminary Frame Classification

On-line object detection: a robotics challenge

ROBO: Robust, Fully Neural Object Detection for Robot Soccer

Dynamic Object Detection and Representation for Mobile Robot Application

Multi-Class Detection and Segmentation of Objects in Depth

Detecting The Objects on The Road Using Modular Lightweight Network

Pan-Tilt Camera and PIR Sensor Fusion Based Moving Object Detection for Mobile Security Robots

High-resolution processing and sigmoid fusion modules for efficient detection of small objects in an embedded system

Real-time Detection of the Moving Object in Video Sequences

Two-Stage Transfer Learning for Heterogeneous Robot Detection and 3D Joint Position Estimation in a 2D Camera Image using CNN