Abstract:In Industry 5.0, Digital Twins bring in flexibility and efficiency for smart manufacturing. Recently, the success of artificial intelligence techniques such as deep learning has led to their adoption in manufacturing and especially in human–robot collaboration. Collaborative manufacturing tasks involving human operators and robots pose significant safety and reliability concerns. In response to these concerns, a deep learning-enhanced Digital Twin framework is introduced through which human operators and robots can be detected and their actions can be classified during the manufacturing process, enabling autonomous decision making by the robot control system. Developed using Unreal Engine 4, our Digital Twin framework complies with the Robotics Operating System specification, and supports synchronous control and communication between the Digital Twin and the physical system. In our framework, a fully-supervised detector based on a faster region-based convolutional neural network is firstly trained on synthetic data generated by the Digital Twin, and then tested on the physical system to demonstrate the effectiveness of the proposed Digital Twin-based framework. To ensure safety and reliability, a semi-supervised detector is further designed to bridge the gap between the twin system and the physical system, and improved performance is achieved by the semi-supervised detector compared to the fully-supervised detector that is simply trained on either synthetic data or real data. The evaluation of the framework in multiple scenarios in which human operators collaborate with a Universal Robot 10 shows that it can accurately detect the human and robot, and classify their actions under a variety of conditions. The data from this evaluation have been made publicly available, and can be widely used for research and operational purposes. Additionally, a semi-automated annotation tool from the Digital Twin framework is published to benefit the collaborative robotics community.

Delay Safety-Aware Digital Twin Empowered Industrial Sensing-Actuation Systems Using Transferable and Reinforced Learning

Digital Twins in Engineering Dynamics: Variational Equation Identification, Feedback Control Design and Their Rapid Update

An Adaptive Evolutionary Framework for the Decision-Making Models of Digital Twin Machining System

Transfer Learning as an Enabler of the Intelligent Digital Twin

Digital Twin Enabled Smart Control Engineering as an Industrial AI: A New Framework and A Case Study

A deep learning-enhanced Digital Twin framework for improving safety and reliability in human–robot collaborative manufacturing

Transfer Learning-enabled Modelling Framework for Digital Twin

An Interpretable Digital Twin for Self-Aware Industrial Machines

Digital Twin-based Intrusion Detection for Industrial Control Systems

A digital twin-based sim-to-real transfer for deep reinforcement learning-enabled industrial robot grasping

A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence

Transfer Learning-based Multiple Digital Twin-assisted Intelligent Mechanical Fault Diagnosis

Intelligent fault diagnosis of machinery using digital twin-assisted deep transfer learning

Adaptive Federated Learning and Digital Twin for Industrial Internet of Things

Digital Twin in Safety-Critical Robotics Applications: Opportunities and Challenges

Digital Twin of Industrial Networked Control System based on Value of Information

A Survey on Digital Twin for Industrial Internet of Things: Applications, Technologies and Tools

Design of Digital Twin Sensing Strategies Via Predictive Modeling and Interpretable Machine Learning

Real-time digital twin-based optimization with predictive simulation learning

Digital-Twin Predictive Control of Nonlinear Systems With Time Delays, Unknown Dynamics, and Communication Delays