Abstract:Multielement decision-making is crucial for the robust deployment of human-robot collaboration (HRC) systems in flexible manufacturing environments with personalized tasks and dynamic scenes. Large Language Models (LLMs) have recently demonstrated remarkable reasoning capabilities in various robotic tasks, potentially offering this capability. However, the application of LLMs to actual HRC systems requires the timely and comprehensive capturing of real-scene information. In this study, we suggest incorporating real scene data into LLMs using digital twin (DT) technology and present a cognitive digital twin prototype system of HRC manipulation, known as HRC-CogiDT. Specifically, we initially construct a scene semantic graph encoding the geometric information of entities, spatial relations between entities, actions of humans and robots, and collaborative activities. Subsequently, we devise a prompt that merges scene semantics with prior knowledge of activities, linking the real scene with LLMs. To evaluate performance, we compile an HRC scene understanding dataset and set up a laboratory-level experimental platform. Empirical results indicate that HRC-CogiDT can swiftly perceive scene changes and make high-level decisions based on varying task requirements, such as task planning, anomaly detection, and schedule reasoning. This study provides promising insights for the future applications of LLMs in robotics. Note to Practitioners—Recently, LLMs have demonstrated significant success in various robotic tasks, suggesting their potential as a powerful tool for robotic decision-making. Motivated by this, to improve the production efficiency of HRC in flexible manufacturing, we innovatively combine LLMs with DT technology, and propose a cognitive DT system for HRC, aiming to integrate LLMs into the decision-making loop of HRC system. Experiments conducted in a laboratory-scale platform indicate that the proposed system can handle different decision-making needs in different HRC activities. This system can provide professional guidance to operators in a comprehensible form and serve as a medium for monitoring the safety and standardization of the manipulation process. Future work will explore the use of virtual space provided by the proposed system to optimize the decision outputs of LLMs to make the proposed system more broadly applicable.

A Digital Twin System for Task-Replanning and Human-Robot Control of Robot Manipulation

Digital Twin-Driven Task Replanning Method for Robot-Environment Physical Interaction

Adaptive Biological Neural Network Control and Virtual Realization for Engineering Manipulator

A Multitasking-Oriented Robot Arm Motion Planning Scheme Based on Deep Reinforcement Learning and Twin Synchro-Control

A digital twin-based frame work for task planning and robot programming in HRC

An Adaptive Path Planning Approach for Digital Twin-Enabled Robot Arm Based on Inverse Kinematics and Deep Reinforcement Learning

Leveraging the Efficiency of Multi-Task Robot Manipulation Via Task-Evoked Planner and Reinforcement Learning

A Digital Twin-Driven Guidance Method for Human-Machine Collaborative Assembly Operations Based on Machine Learning and Computer Vision

Digital Twin-Based Task Rescheduling for Robotic Assembly Line

Toward Cognitive Digital Twin System of Human–Robot Collaboration Manipulation

A novel task-oriented framework for dual-arm robotic assembly task

A Digital Twin System for 6DoF Robot Grasping

Petri Nets-Based Digital Twin Drives Dual-Arm Cooperative Manipulation

A Digital Twin-based environment-adaptive assignment method for human-robot collaboration

A digital twin-driven human–robot collaborative assembly-commissioning method for complex products

A digital twin-based sim-to-real transfer for deep reinforcement learning-enabled industrial robot grasping

A Task-Adaptive Deep Reinforcement Learning Framework for Dual-Arm Robot Manipulation

Digital Twin System for Home Service Robot Based on Motion Simulation

Digital Twin-Driven Human Robot Collaboration Using a Digital Human

Design of a human-robot interaction system for robot teleoperation based on digital twinning

A Novel Approach to Grasping Control of Soft Robotic Grippers based on Digital Twin