Abstract:In Industry 5.0, Digital Twins bring in flexibility and efficiency for smart manufacturing. Recently, the success of artificial intelligence techniques such as deep learning has led to their adoption in manufacturing and especially in human–robot collaboration. Collaborative manufacturing tasks involving human operators and robots pose significant safety and reliability concerns. In response to these concerns, a deep learning-enhanced Digital Twin framework is introduced through which human operators and robots can be detected and their actions can be classified during the manufacturing process, enabling autonomous decision making by the robot control system. Developed using Unreal Engine 4, our Digital Twin framework complies with the Robotics Operating System specification, and supports synchronous control and communication between the Digital Twin and the physical system. In our framework, a fully-supervised detector based on a faster region-based convolutional neural network is firstly trained on synthetic data generated by the Digital Twin, and then tested on the physical system to demonstrate the effectiveness of the proposed Digital Twin-based framework. To ensure safety and reliability, a semi-supervised detector is further designed to bridge the gap between the twin system and the physical system, and improved performance is achieved by the semi-supervised detector compared to the fully-supervised detector that is simply trained on either synthetic data or real data. The evaluation of the framework in multiple scenarios in which human operators collaborate with a Universal Robot 10 shows that it can accurately detect the human and robot, and classify their actions under a variety of conditions. The data from this evaluation have been made publicly available, and can be widely used for research and operational purposes. Additionally, a semi-automated annotation tool from the Digital Twin framework is published to benefit the collaborative robotics community.

Comprehensive Application Analysis of Digital Twin System in Human Robot Collaboration

Human Digital Twin in the Context of Industry 5.0

Digital Twin-Driven Human Robot Collaboration Using a Digital Human

Digital Twins for Human-Robot Collaboration: A Future Perspective

A framework and method for Human-Robot cooperative safe control based on digital twin

Digital Twin for Human–Robot Interactions by Means of Industry 4.0 Enabling Technologies

From Digital Human Modeling to Human Digital Twin: Framework and Perspectives in Human Factors

Human Digital Twin (HDT) Driven Human-Cyber-Physical Systems: Key Technologies and Applications

Digital twins for collaborative robots: A case study in human-robot interaction

A deep learning-enhanced Digital Twin framework for improving safety and reliability in human–robot collaborative manufacturing

Digital twin on concepts, enabling technologies, and applications

Digital Twin in Safety-Critical Robotics Applications: Opportunities and Challenges

Digital Twin in Industry: State-of-the-Art

Human-robot collaboration assembly personalised safety detection method based on digital twin

Comprehensive Survey of the Landscape of Digital Twin Technologies and Their Diverse Applications

Human-Centric Digital Twins in Industry: A Comprehensive Review of Enabling Technologies and Implementation Strategies

Digital Twins in Industry 5.0

Towards a Human-Centric Digital Twin for Human–Machine Collaboration: A Review on Enabling Technologies and Methods

Human Digital Twin: A Survey

Systematic review of digital twin technology and applications

A Digital Twin-based environment-adaptive assignment method for human-robot collaboration