Abstract:The ability to perceive humans is an essential requirement for safe and efficient human-robot interaction. In real-world applications, the need for a robot to interact in real time with multiple humans in a dynamic, 3-D environment presents a significant challenge. The recent availability of commercial color-depth cameras allow for the creation of a system that makes use of the depth dimension, thus enabling a robot to observe its environment and perceive in the 3-D space. Here we present a system for 3-D multiple human perception in real time from a moving robot equipped with a color-depth camera and a consumer-grade computer. Our approach reduces computation time to achieve real-time performance through a unique combination of new ideas and established techniques. We remove the ground and ceiling planes from the 3-D point cloud input to separate candidate point clusters. We introduce the novel information concept, depth of interest, which we use to identify candidates for detection, and that avoids the computationally expensive scanning-window methods of other approaches. We utilize a cascade of detectors to distinguish humans from objects, in which we make intelligent reuse of intermediary features in successive detectors to improve computation. Because of the high computational cost of some methods, we represent our candidate tracking algorithm with a decision directed acyclic graph, which allows us to use the most computationally intense techniques only where necessary. We detail the successful implementation of our novel approach on a mobile robot and examine its performance in scenarios with real-world challenges, including occlusion, robot motion, nonupright humans, humans leaving and reentering the field of view (i.e., the reidentification challenge), human-object and human-human interaction. We conclude with the observation that the incorporation of the depth information, together with the use of modern techniques in new ways, we are able to create an accurate system for real-time 3-D perception of humans by a mobile robot.

Time-of-flight-assisted Kinect camera-based people detection for intuitive human robot cooperation in the surgical operating room

Accurate and Real-Time 3-D Tracking for the Following Robots by Fusing Vision and Ultrasonar Information

The Design and Implementation of Natural Human-Robot Interaction System Based on Kinect Sensor

User-Interactive Robot Skin with Large-Area Scalability for Safer and Natural Human-Robot Collaboration in Future Telehealthcare

A Robotic 3D Perception System for Operating Room Environment Awareness

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Human-robot cooperation using Kinect

Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery

Vision-Based Calibration of Dual RCM-Based Robot Arms in Human-Robot Collaborative Minimally Invasive Surgery

On the Evaluation of Diverse Vision Systems towards Detecting Human Pose in Collaborative Robot Applications

Multi-Camera Multi-Person Tracking and Re-Identification in an Operating Room

Toward safe separation distance monitoring from RGB-D sensors in human-robot interaction

Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery

Real-time multiple human perception with color-depth cameras on a mobile robot

Active Collision Avoidance for Human-Manipulator Safety

Real-time 3D Human Tracking for Mobile Robots with Multisensors

A Auxiliary System Help Patients with Dyskinesia Based on Multi-Camera Stereo Vision

Robust human position estimation in cooperative robotic cells

3D perception from binocular vision for a low cost humanoid robot NAO

Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery

Advancing human-robot collaboration: Predicting operator trajectories through AI and infrared imaging