Abstract:We propose and study a novel cross-reality environment that seamlessly integrates a monoscopic 2D surface (an interactive screen with touch and pen input) with a stereoscopic 3D space (an augmented reality HMD) to jointly host spatial data visualizations. This innovative approach combines the best of two conventional methods of displaying and manipulating spatial 3D data, enabling users to fluidly explore diverse visual forms using tailored interaction techniques. Providing such effective 3D data exploration techniques is pivotal for conveying its intricate spatial structures -- often at multiple spatial or semantic scales -- across various application domains and requiring diverse visual representations for effective visualization. To understand user reactions to our new environment, we began with an elicitation user study, in which we captured their responses and interactions. We observed that users adapted their interaction approaches based on perceived visual representations, with natural transitions in spatial awareness and actions while navigating across the physical surface. Our findings then informed the development of a design space for spatial data exploration in cross-reality. We thus developed cross-reality environments tailored to three distinct domains: for 3D molecular structure data, for 3D point cloud data, and for 3D anatomical data. In particular, we designed interaction techniques that account for the inherent features of interactions in both spaces, facilitating various forms of interaction, including mid-air gestures, touch interactions, pen interactions, and combinations thereof, to enhance the users' sense of presence and engagement. We assessed the usability of our environment with biologists, focusing on its use for domain research. In addition, we evaluated our interaction transition designs with virtual and mixed-reality experts to gather further insights.

ARPad: Compound Spatial-Semantic Gesture Interaction in Augmented Reality.

Eliciting Multimodal Gesture+Speech Interactions in a Multi-Object Augmented Reality Environment

Semantic Augmented Reality Environment with Material-Aware Physical Interactions

A touchable virtual screen interaction system with handheld Kinect camera

Interaction With Gaze, Gesture, and Speech in a Flexibly Configurable Augmented Reality System

Gesture-based target acquisition in virtual and augmented reality.

An interaction system using mixed hand gestures.

Interaction of Spatial Computing In Augmented Reality

A novel AR remote collaborative platform for sharing 2.5D gestures and gaze

Elicitation and Evaluation of Hand-based Interaction Language for 3D Conceptual Design in Mixed Reality

Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI

Remote Object Taking and Movement in Augment Reality

A mixed reality platform for assembly assistance based on gaze interaction in industry

Augmenting collaborative interaction with shared visualization of eye movement and gesture in VR

GesMoSketch: A System for 3D Sketching in AR with One Mobile Device

Augmented reality spatial programming paradigm applied to end-user robot programming

A gesture- and head-based multimodal interaction platform for MR remote collaboration

Collaborative programming problem-solving in augmented reality: Multimodal analysis of effectiveness and group collaboration

Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents

Hands and speech in space: multimodal interaction with augmented reality interfaces

SpatialTouch: Exploring Spatial Data Visualizations in Cross-reality