Abstract:With the rapid development of mixed reality (MR) technology, many compact, lightweight, and powerful devices suitable for remote collaboration, such as MR headsets, hand trackers, and 3D cameras, become readily available, providing hardware and software support for remote collaboration. Consequently, exploring MR technologies for remote collaboration on physical industry tasks is becoming increasingly worthwhile. In many complex production scenarios, such as assembly tasks, significant gains can be achieved by having remote experts assist local workers to manipulate objects in local workspaces. However, it can be challenging for a remote expert to carry out effective spatial reference and action demonstration in a local scene. Sharing 3D stereoscopic scenes can provide depth perception and support remote experts to move and explore a local user's environment freely. Previous studies have demonstrated that gesture-based interaction is natural and intuitive, and interaction based on virtual replicas can provide clear guidance, especially for industrial physical tasks. In this study, we develop an MR remote collaboration system that shares the stereoscopic scene of the local workspace by using real-time 3D video. This system combines gesture cues and virtual replicas in a complementary manner to support the remote expert to create augmented reality (AR) guidance for the local worker naturally and intuitively in the virtual reality immersive space. A formal user study was performed to explore the effects of two different modalities interface in industrial assembly tasks: our novel method of using the combination of virtual replicas and gesture cues in the 3D video (VG3DV), and a method similar to the popular method currently of using gesture cues in the 3D video (G3DV). We found that using the VG3DV can significantly improve the performance and user experience of MR remote collaboration in industrial assembly tasks. Finally, some conclusions and future research directions were given.

Live Mixed Reality Video Production for Educational Presentation

Stereo Camera System and Its Application in Networked Virtual Environment

A touchable virtual screen interaction system with handheld Kinect camera

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

Real-time Depth Extraction and Multi-View Rendering Algorithm Based on Kinect

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

Real time stereo rendering for augmented reality on 3DTV system

Real-Time 3D Model Reconstruction and Interaction Using Kinect for a Game-Based Virtual Laboratory

Live Video Montage with a Rotating Camera

Hybrid-MVS: Robust Multi-View Reconstruction with Hybrid Optimization of Visual and Depth Cues

Real-time 3d reconstruction of dynamic scenes with multiple kinect v2 sensors

Depth-color fusion strategy for 3-D scene modeling with Kinect

Kinect based 3D video generation

Live Three-Dimensional Content for Augmented Reality.

Inserting Virtual Characters into Live Video of Real Scenes

A novel animation authoring framework for the virtual teacher performing experiment in mixed reality

Virtual-Real Fusion with Dynamic Scene from Videos

Application of Physical Interactive Mixed Reality System Based on MLAT in the Field of Stage Performance

Real-time 3D Human Motion Capture from Binocular Stereo Vision and Its Use on Human Body Animation

Bringing Telepresence to Every Desk

Real-time 3D video-based MR remote collaboration using gesture cues and virtual replicas