Abstract:Multielement decision-making is crucial for the robust deployment of human-robot collaboration (HRC) systems in flexible manufacturing environments with personalized tasks and dynamic scenes. Large Language Models (LLMs) have recently demonstrated remarkable reasoning capabilities in various robotic tasks, potentially offering this capability. However, the application of LLMs to actual HRC systems requires the timely and comprehensive capturing of real-scene information. In this study, we suggest incorporating real scene data into LLMs using digital twin (DT) technology and present a cognitive digital twin prototype system of HRC manipulation, known as HRC-CogiDT. Specifically, we initially construct a scene semantic graph encoding the geometric information of entities, spatial relations between entities, actions of humans and robots, and collaborative activities. Subsequently, we devise a prompt that merges scene semantics with prior knowledge of activities, linking the real scene with LLMs. To evaluate performance, we compile an HRC scene understanding dataset and set up a laboratory-level experimental platform. Empirical results indicate that HRC-CogiDT can swiftly perceive scene changes and make high-level decisions based on varying task requirements, such as task planning, anomaly detection, and schedule reasoning. This study provides promising insights for the future applications of LLMs in robotics. Note to Practitioners—Recently, LLMs have demonstrated significant success in various robotic tasks, suggesting their potential as a powerful tool for robotic decision-making. Motivated by this, to improve the production efficiency of HRC in flexible manufacturing, we innovatively combine LLMs with DT technology, and propose a cognitive DT system for HRC, aiming to integrate LLMs into the decision-making loop of HRC system. Experiments conducted in a laboratory-scale platform indicate that the proposed system can handle different decision-making needs in different HRC activities. This system can provide professional guidance to operators in a comprehensible form and serve as a medium for monitoring the safety and standardization of the manipulation process. Future work will explore the use of virtual space provided by the proposed system to optimize the decision outputs of LLMs to make the proposed system more broadly applicable.

An LLM-based approach for enabling seamless Human-Robot collaboration in assembly

Autonomous Human-Robot Collaborative Assembly Method Driven by the Fusion of Large Language Model and Digital Twin

Enhancing Human-Robot Collaborative Assembly in Manufacturing Systems Using Large Language Models

An LLM-based vision and language cobot navigation approach for Human-centric Smart Manufacturing

Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration

LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks

Proactive Human-Robot Co-Assembly: Leveraging Human Intention Prediction and Robust Safe Control

A vision-language-guided robotic action planning approach for ambiguity mitigation in human–robot collaborative manufacturing

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

Human-robot collaboration in disassembly for sustainable manufacturing

Toward Cognitive Digital Twin System of Human–Robot Collaboration Manipulation

Enabling collaborative assembly between humans and robots using a digital twin system

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

A Cooperative Shared Control Scheme Based on Intention Recognition for Flexible Assembly Manufacturing

Towards seamless human robot collaboration: integrating multimodal interaction

Natural language instructions for intuitive human interaction with robotic assistants in field construction work

A Survey of Robot Learning Strategies for Human-Robot Collaboration in Industrial Settings

Extended Reality for Enhanced Human-Robot Collaboration: a Human-in-the-Loop Approach

Integrating Large Language Models with Multimodal Virtual Reality Interfaces to Support Collaborative Human-Robot Construction Work

A digital twin-driven human–robot collaborative assembly-commissioning method for complex products

HRC for dual-robot intelligent assembly system based on multimodal perception