Abstract:We propose and study a novel cross-reality environment that seamlessly integrates a monoscopic 2D surface (an interactive screen with touch and pen input) with a stereoscopic 3D space (an augmented reality HMD) to jointly host spatial data visualizations. This innovative approach combines the best of two conventional methods of displaying and manipulating spatial 3D data, enabling users to fluidly explore diverse visual forms using tailored interaction techniques. Providing such effective 3D data exploration techniques is pivotal for conveying its intricate spatial structures -- often at multiple spatial or semantic scales -- across various application domains and requiring diverse visual representations for effective visualization. To understand user reactions to our new environment, we began with an elicitation user study, in which we captured their responses and interactions. We observed that users adapted their interaction approaches based on perceived visual representations, with natural transitions in spatial awareness and actions while navigating across the physical surface. Our findings then informed the development of a design space for spatial data exploration in cross-reality. We thus developed cross-reality environments tailored to three distinct domains: for 3D molecular structure data, for 3D point cloud data, and for 3D anatomical data. In particular, we designed interaction techniques that account for the inherent features of interactions in both spaces, facilitating various forms of interaction, including mid-air gestures, touch interactions, pen interactions, and combinations thereof, to enhance the users' sense of presence and engagement. We assessed the usability of our environment with biologists, focusing on its use for domain research. In addition, we evaluated our interaction transition designs with virtual and mixed-reality experts to gather further insights.

Multi-Depth-Camera Sensing and Interaction in Smart Space.

User-Interactive Robot Skin with Large-Area Scalability for Safer and Natural Human-Robot Collaboration in Future Telehealthcare

A touchable virtual screen interaction system with handheld Kinect camera

Context Awareness, the Spirit of Pervasive Computing

Privacy-preserving Activity Recognition Using Multimodal Sensors in Smart Office

Human-centric Application in Cyber-Physical System: an Inertial-based Motion Capture and Recognition System

Multi-sensor Fusion for Interactive Visual Computing in Mixed Environment.

Motion-Based Perceptual User Interface

Touch-Driven Interaction between Physical Space and Cyberspace with NFC.

A Penetrable Interactive 3D Display Based on Motion Recognition （invited Paper）

Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption

MicroCam: Leveraging Smartphone Microscope Camera for Context-Aware Contact Surface Sensing

Digital Proxemics: Designing Social and Collaborative Interaction in Virtual Environments

Panoptic Studio: A Massively Multiview System for Social Interaction Capture

Neighbor-Environment Observer: An Intelligent Agent for Immersive Working Companionship

Pull and Push: Proximity-Aware User Interface for Navigating in 3D Space Using a Handheld Camera

An Agent-Based Intelligent HCI Information System in Mixed Reality

SpatialTouch: Exploring Spatial Data Visualizations in Cross-reality

A multisensory Interaction Framework for Human-Cyber–Physical System based on Graph Convolutional Networks

Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI

Multi-View Active Sensing for Human-Robot Interaction via Hierarchically Connected Tree