Abstract:In Industry 5.0, Digital Twins bring in flexibility and efficiency for smart manufacturing. Recently, the success of artificial intelligence techniques such as deep learning has led to their adoption in manufacturing and especially in human–robot collaboration. Collaborative manufacturing tasks involving human operators and robots pose significant safety and reliability concerns. In response to these concerns, a deep learning-enhanced Digital Twin framework is introduced through which human operators and robots can be detected and their actions can be classified during the manufacturing process, enabling autonomous decision making by the robot control system. Developed using Unreal Engine 4, our Digital Twin framework complies with the Robotics Operating System specification, and supports synchronous control and communication between the Digital Twin and the physical system. In our framework, a fully-supervised detector based on a faster region-based convolutional neural network is firstly trained on synthetic data generated by the Digital Twin, and then tested on the physical system to demonstrate the effectiveness of the proposed Digital Twin-based framework. To ensure safety and reliability, a semi-supervised detector is further designed to bridge the gap between the twin system and the physical system, and improved performance is achieved by the semi-supervised detector compared to the fully-supervised detector that is simply trained on either synthetic data or real data. The evaluation of the framework in multiple scenarios in which human operators collaborate with a Universal Robot 10 shows that it can accurately detect the human and robot, and classify their actions under a variety of conditions. The data from this evaluation have been made publicly available, and can be widely used for research and operational purposes. Additionally, a semi-automated annotation tool from the Digital Twin framework is published to benefit the collaborative robotics community.

System Research on the Efficient Identification of Failures or Defects in Collaborative Robots Using Digital Twin Technology

A Hybrid Digital Twin Scheme for the Condition Monitoring of Industrial Collaborative Robots

Addressing Non-Intervention Challenges via Resilient Robotics utilizing a Digital Twin

Digital Twin in Safety-Critical Robotics Applications: Opportunities and Challenges

A deep learning-enhanced Digital Twin framework for improving safety and reliability in human–robot collaborative manufacturing

Comprehensive Application Analysis of Digital Twin System in Human Robot Collaboration

Digital Twin of Industrial Robots Over the Cloud: Framework, Design and Analysis

A Digital Twin Based Large-Area Robot Skin System for Safer Human-Centered Healthcare Robots Toward Healthcare 4.0

A framework and method for Human-Robot cooperative safe control based on digital twin

A digital twin–driven monitoring framework for dual-robot collaborative manipulation

Digital Twin-Driven Human Robot Collaboration Using a Digital Human

Human-Robot Collision Avoidance Scheme for Industrial Settings Based on Injury Classification

A Modular Digital Twinning Framework for Safety Assurance of Collaborative Robotics

Embedding Digital Twin Technology in Robotics

A Distributed Computing Real-Time Safety System of Collaborative Robot

Digital Twin: Concept of Hybrid Programming for Industrial Robots — Use Case

A Digital Twin-based environment-adaptive assignment method for human-robot collaboration

Digital Twin of Intelligent Small Surface Defect Detection with Cyber-Manufacturing Systems

A Data-driven Approach Towards Human-robot Collaborative Problem Solving in a Shared Space

A Safety and Collision Avoidance System for Industrial Robots

A Digital Twinning Platform for Integrated Sensing, Communications and Robotics