Abstract:With the rapid development of mixed reality (MR) technology, many compact, lightweight, and powerful devices suitable for remote collaboration, such as MR headsets, hand trackers, and 3D cameras, become readily available, providing hardware and software support for remote collaboration. Consequently, exploring MR technologies for remote collaboration on physical industry tasks is becoming increasingly worthwhile. In many complex production scenarios, such as assembly tasks, significant gains can be achieved by having remote experts assist local workers to manipulate objects in local workspaces. However, it can be challenging for a remote expert to carry out effective spatial reference and action demonstration in a local scene. Sharing 3D stereoscopic scenes can provide depth perception and support remote experts to move and explore a local user's environment freely. Previous studies have demonstrated that gesture-based interaction is natural and intuitive, and interaction based on virtual replicas can provide clear guidance, especially for industrial physical tasks. In this study, we develop an MR remote collaboration system that shares the stereoscopic scene of the local workspace by using real-time 3D video. This system combines gesture cues and virtual replicas in a complementary manner to support the remote expert to create augmented reality (AR) guidance for the local worker naturally and intuitively in the virtual reality immersive space. A formal user study was performed to explore the effects of two different modalities interface in industrial assembly tasks: our novel method of using the combination of virtual replicas and gesture cues in the 3D video (VG3DV), and a method similar to the popular method currently of using gesture cues in the 3D video (G3DV). We found that using the VG3DV can significantly improve the performance and user experience of MR remote collaboration in industrial assembly tasks. Finally, some conclusions and future research directions were given.

Research on Gesture Recognition and Interaction of Virtual Collaborative Disassembly Training

MIVAS: A Multi-Modal Immersive Virtual Assembly System

A Natural Bare-Hand Interaction Method With Augmented Reality for Constraint-Based Virtual Assembly

An interaction system using mixed hand gestures.

Research On Aircraft Virtual Assembly Technology Based On Gesture Recognition

A natural bare-hand interface-enabled interactive AR assembly guidance

Constrained Behavior Manipulation for Interactive Assembly in a Virtual Environment

Interactive task planning in virtual assembly.

A multimodal interface to situated assembly robot systems

Interaction and control with the auxiliary of hand gesture

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

Real-time 3D video-based MR remote collaboration using gesture cues and virtual replicas

Virtual Manipulation in an Immersive Virtual Environment: Simulation of Virtual Assembly

Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction

An Integrated Virtual Design And Virtual Assembly Environment

Grasp Identification and Multi-Finger Haptic Feedback for Virtual Assembly

Research into Virtual Hand Perception Mechanism Orienting Virtual Assembly Environment

Construction of feature-matching perception in virtual assembly

Agricultural Machinery Virtual Assembly System Using Dynamic Gesture Recognitive Interaction Based on a CNN and LSTM Network

Research on Immersive Virtual Assembly Technology

Gesture-driven interaction service system for complex operations in digital twin manufacturing cells