A novel MR remote collaborative assembly system using reconstructed attribute-enhanced product models

Quan Yu,Shuxia Wang,Qiang Zhang,Xiangyu Zhang,Yuxiang Yan,Liyuan Shao

DOI: https://doi.org/10.1007/s00170-024-14721-2

IF: 3.563

2024-11-03

The International Journal of Advanced Manufacturing Technology

Abstract:In mixed reality (MR) remote collaboration systems used for assembly tasks, the product models, serving as digital representations of parts, are pivotal in facilitating users' comprehension of assembly objects, enabling interactive operations, and demonstration guidance in a virtual-real fusion environment. These product models represent the most critical virtual assets. However, the existing system's product model preparation cycle is lengthy, and its actual application performance is poor, with insufficient support capabilities, rendering it unsuitable for urgent assembly tasks with limited information. In this study, we developed a novel MR remote collaborative assembly system, distinguished by several key innovations. Firstly, we devised an efficient scheme for constructing product models within the system, enabling users to rapidly create 3D product models of parts for assembly without assembly task-related prior information, thus significantly reducing costs and resource demands. Secondly, we defined and summarized for the first time the essential MR remote collaborative assembly attributes that product models should incorporate. These attributes were integrated into each product model during construction to enhance functionality and usability in collaborative assembly tasks. Using the proposed system, users scan the workspace with a handheld camera to capture parts slated for assembly. The collected data undergoes processes such as scene reconstruction, instance segmentation, and instance completion to generate corresponding 3D virtual models for each part. Furthermore, enriching these models with comprehensive MR remote collaborative assembly attributes, including realistic attributes, interactive attributes, spatial attributes, and virtual assembly attributes, results in attribute-enhanced product models. To evaluate the performance of these attribute-enhanced product models in remote collaborative physical assembly tasks, we conducted a user study. The results demonstrate that in collaborative assembly scenarios with limited information, product models with MR remote collaborative assembly attributes exhibit better user usability and task support compared to traditional solutions, which can significantly improve collaborative assembly efficiency, reduce operational errors, decrease users' workload, and provide a superior user experience.

engineering, manufacturing,automation & control systems