VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual Replicas

Xincheng Huang,Michael Yin,Ziyi Xia,Robert Xiao

2024-08-06

Abstract:Asymmetric AR/VR collaboration systems bring a remote VR user to a local AR user's physical environment, allowing them to communicate and work within a shared virtual/physical space. Such systems often display the remote environment through 3D reconstructions or 360-degree videos. While 360-degree cameras stream an environment in higher quality, they lack spatial information, making them less interactable. We present VirtualNexus, an AR/VR collaboration system that enhances 360-degree video AR/VR collaboration with environment cutouts and virtual replicas. VR users can define cutouts of the remote environment to interact with as a world-in-miniature, and their interactions are synchronized to the local AR perspective. Furthermore, AR users can rapidly scan and share 3D virtual replicas of physical objects using neural rendering. We demonstrated our system's utility through 3 example applications and evaluated our system in a dyadic usability test. VirtualNexus extends the interaction space of 360-degree telepresence systems, offering improved physical presence, versatility, and clarity in interactions.

Human-Computer Interaction

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the limitations of existing 360-degree videos in augmented reality (AR) and virtual reality (VR) remote collaboration. Specifically, while 360-degree videos provide a high-quality visual experience, they lack spatial information, making it difficult for users to engage in efficient two-way interactions. Additionally, virtual objects can only float in front of the 360-degree video and cannot interact authentically with the physical environment, which undermines the user's sense of physical presence.

To overcome these limitations, the authors propose a system called VirtualNexus, which enhances 360-degree video AR/VR collaboration by introducing environment cutouts and virtual replicas. Specifically:

1. **Environment Cutouts**:
- Allow remote VR users to select a portion of the 360-degree environment and interact with it as a World in Miniature (WiM).
- These cutouts can be zoomed in, scaled, and rotated to provide precise control.
- Any changes within the cutouts are synchronized with the original 360-degree video and the AR user's physical environment.

2. **Virtual Replicas**:
- Allow local AR users to quickly scan physical objects and generate shared virtual replicas.
- Using Neural Radiance Fields (NeRF) technology, high-quality virtual replicas can be created within 1-3 minutes.
- Virtual replicas can be used to enhance interactions between the physical and virtual environments.

Through these innovative features, VirtualNexus expands the interactive space of 360-degree video remote collaboration, enhancing physical presence, diversity, and interaction clarity.