Abstract:Sorting tasks are typical applications in the product and industrial domains. When facing new settings, such as different types of objects and their positions, the system has to be reprogrammed by experienced engineers, which decreases production efficiency and increases labour and downtime costs. In this paper, an off-line programming framework for sorting tasks based on human demonstration has been developed, taking advantage of machine vision and programming-by-demonstration concepts. Only one eye-to-hand depth camera has been used to record both the demonstrated trajectory and the corresponding relationship of the sorting task. A feature generalization algorithm has also been proposed to ensure that the system is able to reproduce the sorting task using both the key features of the trajectory and the sorting features. Finally, tracking, object and target matching, and generalization experiments have been conducted using the JAKA Zu7 robot with different types of fruit models and target boxes. The results show that the proposed framework is able to record the sorting trajectory at a frequency of 10 Hz, acquire and maintain the sorting relationship after new demonstration, and automatically reproduce the sorting task in new settings. Note to Practitioners —This paper is motivated by a goal to accomplish intuitive and user-friendly off-line programming for sorting task. The programming framework should be usable by non-experts and be able to complete programming tasks with a few demonstrations when faced with new scenarios, such as different types of objects, different correspondence relationships of the object, and different positions of the object. By observing the human demonstrated sorting tasks, the above-mentioned features can be effectively extracted, so as to complete the convenient teaching. The underlying principle of this paper is to let the robot observe human demonstrated sorting task by a depth camera, tracking the relative trajectory between the object and box as well as recording their matching information. The key features of the trajectory are extracted and further generalized combing the matching information to automatically reproduce the executable path in different settings. The proposed framework allows the users without prior programming knowledge to program the sorting task in new setting by simply conducting the pick and place process with their hands.

Robot teaching playback based on machine vision and human hand demonstration

Industrial Robot Programming by Demonstration

Perception of Demonstration for Automatic Programing of Robotic Assembly: Framework, Algorithm, and Validation

View-based teaching/playback for robotic manipulation

An Online Robot Teaching Method using Static Hand Gestures and Poses

Pomdp Based Robot Teaching for High Precision Assembly in Manufacturing Automation

A Feature Reserved Teaching Method for Pick-Place System under Robot Operating System

Online Robot Teaching with Natural Human–Robot Interaction

A Task-Learning Strategy for Robotic Assembly Tasks from Human Demonstrations

Intuitive Robot Teaching By Hand Guided Demonstration

Motion capture and AR based programming by demonstration for industrial robots using handheld teaching device

Application of Virtual Reality Technology in Teaching and Training System of Processing Industrial Robot

Natural Teaching for Humanoid Robot Via Human-in-the-loop Scene-Motion Cross-Modal Perception.

Vision-Based Iterative Trajectory Tracking for Industrial Manipulators

Robot Teaching by Teleoperation Based on Visual Interaction and Neural Network Learning

A Robot Learning from Demonstration Method Based on Neural Network and Teleoperation

Human–robot Skill Transmission for Mobile Robot Via Learning by Demonstration

Off-Line Programming Framework for Sorting Task Based on Human-Demonstration

A Method Based on Wearable Devices for Controlling Teaching of Robots for Human-robot Collaboration

A Human–Robot Collaboration Method Using a Pose Estimation Network for Robot Learning of Assembly Manipulation Trajectories From Demonstration Videos

Robot Teaching by Teleoperation Based on Visual Interaction and Extreme Learning Machine