Abstract:With the rapid development of mixed reality (MR) technology, many compact, lightweight, and powerful devices suitable for remote collaboration, such as MR headsets, hand trackers, and 3D cameras, become readily available, providing hardware and software support for remote collaboration. Consequently, exploring MR technologies for remote collaboration on physical industry tasks is becoming increasingly worthwhile. In many complex production scenarios, such as assembly tasks, significant gains can be achieved by having remote experts assist local workers to manipulate objects in local workspaces. However, it can be challenging for a remote expert to carry out effective spatial reference and action demonstration in a local scene. Sharing 3D stereoscopic scenes can provide depth perception and support remote experts to move and explore a local user's environment freely. Previous studies have demonstrated that gesture-based interaction is natural and intuitive, and interaction based on virtual replicas can provide clear guidance, especially for industrial physical tasks. In this study, we develop an MR remote collaboration system that shares the stereoscopic scene of the local workspace by using real-time 3D video. This system combines gesture cues and virtual replicas in a complementary manner to support the remote expert to create augmented reality (AR) guidance for the local worker naturally and intuitively in the virtual reality immersive space. A formal user study was performed to explore the effects of two different modalities interface in industrial assembly tasks: our novel method of using the combination of virtual replicas and gesture cues in the 3D video (VG3DV), and a method similar to the popular method currently of using gesture cues in the 3D video (G3DV). We found that using the VG3DV can significantly improve the performance and user experience of MR remote collaboration in industrial assembly tasks. Finally, some conclusions and future research directions were given.

Study of efficiency of multi view system in multi-disciplinary collaboration task

Role Based Cross-Project Collaboration in Multiple Distributed Software Design Projects

Effects of Multiple Spatial Perspectives and Mutual Awareness on Collaborative Object Manipulation

Multi-View Clustering Via Simultaneously Learning Shared Subspace And Affinity Matrix

A Streaming-Based Approach for Remote Interaction of the Multi-Channel Display System for Group Users

Collaboration Upon Heterogeneous Platforms - From Desktop Pc To Handheld Device

Viewpoint-Controllable Telepresence: A Robotic-Arm-Based Mixed-Reality Telecollaboration System

A Multi-modal Virtual-Real Fusion System for Multi-task Human-Computer Interaction.

Exploring Configurations for Multi-user Communication in Virtual Reality

Development of a Synchronized Human-Robot-Virtuality Interaction System Using Cooperative Robot and Motion Capture Device

Real-time 3D video-based MR remote collaboration using gesture cues and virtual replicas

Multimodal Interfaces for Effective Teleoperation

Development of a Three-Dimensional Multimode Visual Immersive System With Applications in Telepresence

An Internet-Based Conference System for Real-Time Distributed Design Evaluation

Real-time multimodal interaction in virtual reality - a case study with a large virtual interface

Architectural design creativity in Multi-User Virtual Environment: A comparative analysis between remote collaboration media

Collaborative MR Workspace with Shared 3D Vision Based on Stereo Video Transmission

A large display-based approach supporting natural user interaction in virtual reality environment

An evaluation of multimodal interaction techniques for 3D layout constraint solver in a desktop-based virtual environment

Desk2Desk: Optimization-based Mixed Reality Workspace Integration for Remote Side-by-side Collaboration

A gesture- and head-based multimodal interaction platform for MR remote collaboration