Abstract:We propose and study a novel cross-reality environment that seamlessly integrates a monoscopic 2D surface (an interactive screen with touch and pen input) with a stereoscopic 3D space (an augmented reality HMD) to jointly host spatial data visualizations. This innovative approach combines the best of two conventional methods of displaying and manipulating spatial 3D data, enabling users to fluidly explore diverse visual forms using tailored interaction techniques. Providing such effective 3D data exploration techniques is pivotal for conveying its intricate spatial structures -- often at multiple spatial or semantic scales -- across various application domains and requiring diverse visual representations for effective visualization. To understand user reactions to our new environment, we began with an elicitation user study, in which we captured their responses and interactions. We observed that users adapted their interaction approaches based on perceived visual representations, with natural transitions in spatial awareness and actions while navigating across the physical surface. Our findings then informed the development of a design space for spatial data exploration in cross-reality. We thus developed cross-reality environments tailored to three distinct domains: for 3D molecular structure data, for 3D point cloud data, and for 3D anatomical data. In particular, we designed interaction techniques that account for the inherent features of interactions in both spaces, facilitating various forms of interaction, including mid-air gestures, touch interactions, pen interactions, and combinations thereof, to enhance the users' sense of presence and engagement. We assessed the usability of our environment with biologists, focusing on its use for domain research. In addition, we evaluated our interaction transition designs with virtual and mixed-reality experts to gather further insights.

Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together

Spatial Assisted Human-Drone Collaborative Navigation and Interaction through Immersive Mixed Reality

Spatial Computing: Defining the Vision for the Future

Mobile Mixed-Reality Interfaces That Enhance Human–Robot Interaction in Shared Spaces

Investigating Mixed Reality for Communication Between Humans and Mobile Manipulators

Mixed Reality as Communication Medium for Human-Robot Collaboration

Interaction of Spatial Computing In Augmented Reality

Recent Development of Human-Robot Natural Interaction in Spatial Cognition Tasks

Unveiling the Era of Spatial Computing

Spatial Awareness in Pervasive Ecosystems

Immersive Robot Teleoperation Based on User Gestures in Mixed Reality Space

Mixed Reality Spatial Computing in a Remote Learning Classroom

The Design of Human-Robot Spatial Cognition Interaction System

SpatialTouch: Exploring Spatial Data Visualizations in Cross-reality

Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design

Mixed reality-integrated 3D/2D vision mapping for intuitive teleoperation of mobile manipulator

A 3D Mixed Reality Interface for Human-Robot Teaming

Immersive Assistance System for Intuitive Robot Programming using Mixed-Reality and Digital Twin

Amplifying robotics capacities with a human touch: An immersive low-latency panoramic remote system

Learning robot motor skills with mixed reality

Spatial Computing: Concept, Applications, Challenges and Future Directions