Abstract:The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.

Indoor environment applications for mobile robots using Kinect2.0

The Design and Implementation of Natural Human-Robot Interaction System Based on Kinect Sensor

Design of Kinect-Based Human Robot Interaction Systems for A Robocup Middle Size League Soccer Robot

Design and Implementation of an Intelligent Passenger Counting System Based on the Kinect Sensor

Online People Tracking and Identification with RFID and Kinect

Kinect v2 for mobile robot navigation: Evaluation and modeling

RGB-D Human Identification and Tracking in a Smart Environment

A novel target detection and localization method in indoor environment for mobile robot based on improved YOLOv5

Mobile Robot Indoor Positioning Based on a Combination of Visual and Inertial Sensors

Using a Kinect Sensor to Acquire Biological Motion: Toolbox and Evaluation.

Hybrid Visual Natural Landmark–based Localization for Indoor Mobile Robots

Indoor 3D Video Monitoring Using Multiple Kinect Depth-Cameras

Characterization of a Multi-User Indoor Positioning System Based on Low Cost Depth Vision (Kinect) for Monitoring Human Activity in a Smart Home

Indoor Human Action Recognition Based on Dual Kinect V2 and Improved Ensemble Learning Method

Human-centric Application in Cyber-Physical System: an Inertial-based Motion Capture and Recognition System

Human Motion Tracking by Multiple RGBD Cameras.

Human-computer interaction system for hand recognition based on depth information from Kinect sensor

Real-time multiple human perception with color-depth cameras on a mobile robot

Robust Depth-Aided RGBD-Inertial Odometry for Indoor Localization

Skeletal Data Matching and Merging from Multiple RGB-D Sensors for Room-Scale Human Behaviour Tracking

Research of SLAM for Indoor Environment based on Kinect