Abstract:Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth introduction of cobots into an industrial assembly line is described, with a view to performing actions in parallel with the human operators and enabling interaction between them. The proposed active vision system uses two RGB-D cameras that record different points of view of gestures and poses of the operator, to build an external perception layer for the robot that facilitates spatiotemporal adaptation, in accordance with the human's behavior. The use-case of this work is concerned with LCD TV assembly of an appliance manufacturer, comprising of two parts. The first part of the above-mentioned operation is assigned to a robot, strengthening the assembly line. The second part is assigned to a human operator. Gesture recognition, pose estimation, physical interaction, and sonic notification, create a multimodal human-robot interaction system. Five experiments are performed, to test if gesture recognition and pose estimation can reduce the cycle time and range of motion of the operator, respectively. Physical interaction is achieved using the force sensor of the cobot. Pose estimation through a skeleton-tracking algorithm provides the cobot with human pose information and makes it spatially adjustable. Sonic notification is added for the case of unexpected incidents. A real-time gesture recognition module is implemented through a Deep Learning architecture consisting of Convolutional layers, trained in an egocentric view and reducing the cycle time of the routine by almost 20%. This constitutes an added value in this work, as it affords the potential of recognizing gestures independently of the anthropometric characteristics and the background. Common metrics derived from the literature are used for the evaluation of the proposed system. The percentage of spatial adaptation of the cobot is proposed as a new KPI for a collaborative system and the opinion of the human operator is measured through a questionnaire that concerns the various affective states of the operator during the collaboration.

CobotAR: Interaction with Robots using Omnidirectionally Projected Image and DNN-based Gesture Recognition

CobotGear: Interaction with Collaborative Robots using Wearable Optical Motion Capturing Systems

Visualizing Robot Intent for Object Handovers with Augmented Reality

ARROCH: Augmented Reality for Robots Collaborating with a Human

A Novel Gesture Recognition System for Intelligent Interaction with a Nursing-Care Assistant Robot

Enabling Intuitive Human-Robot Teaming Using Augmented Reality and Gesture Control

Augmented Reality for Interactive Robot Control

Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots

Development of a Human–Robot Interface for Cobot Trajectory Planning Using Mixed Reality

Design and Evaluation of an Augmented Reality Head-Mounted Display Interface for Human Robot Teams Collaborating in Physically Shared Manufacturing Tasks

ArUcoGlide: a Novel Wearable Robot for Position Tracking and Haptic Feedback to Increase Safety During Human-Robot Interaction

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Real-Time Multi-modal Human-Robot Collaboration Using Gestures and Speech

Arm Robot: AR-Enhanced Embodied Control and Visualization for Intuitive Robot Arm Manipulation

AR-Enabled Persistent Human-Machine Interfaces via a Scalable Soft Electrode Array

Amplifying robotics capacities with a human touch: An immersive low-latency panoramic remote system

Evaluating the Augmented Reality Human-Robot Collaboration System

A Novel Robot Teaching System Based on Augmented Reality

Non-invasive Cognitive-level Human Interfacing for the Robotic Restoration of Reaching & Grasping

A Mixed-Reality Tele-Operation Method for High-Level Control of a Legged-Manipulator Robot

Toward Programming a Collaborative Robot by Interacting with Its Digital Twin in a Mixed Reality Environment