Abstract:Physical face-to-face collaboration with someone gives a higher-quality experience compared to mediated communication options, such as a phone- or video-based chat. Participants can share rich sensory cues to multiple human senses in a physical space. Also, the perceptual sensing of the surrounding environment including other peoples' reactions can influence human communication and emotion, and thus collaborative performance. Shared spaces in virtual environments provide degraded sensory experiences because most commercial virtual reality systems typically provide only visual and audio feedback. The impact of richer, multi-sensory feedback on joint decision-making tasks in VR is still an open area of research. Two independent studies exploring this topic are presented in this paper. We implemented a multi-sensory system that delivers vision, audio, tactile, and smell feedback, and we compared the system to a typical VR system. The scenario placed two users in a virtual theme-park safari ride with a number of non-player character (NPC) passengers to simulate realistic scenarios compared to the real-world and we varied the type and complexity of NPCs reactions to participants. In Experiment 1, we provided both users with either multi-sensory or typical sensory feedback symmetrically as a between-subjects factor, and used NPC reaction type as a within-subjects factor. In Experiment 2, we provided sensory feedback asymmetrically to each user (i.e., one had multi-sensory cues and the other had typical sensory cues) as a between-subjects factor, and used NPC reaction type as a within-subjects factor. We found that the number of sensory channels and NPC reactions did not influence user perception significantly under either symmetric or asymmetric sensory feedback conditions. However, after accounting for individual personality traits (e.g., assertive, passive), as well as any existing relationship between the pairs, we found that increasing the number of sensory channels can significantly improve subjective responses.

Exploring Subjective and Objective Performance of Multimodal Interactions in Different Physical Environments.

Multimodal Interfaces for Effective Teleoperation

Review of Sensory Feedback Simulation Methods in Multi-modal Human-computer Interaction

Exploring Data Input Problems in Mixed Reality Environments: Proposal and Evaluation of Natural Interaction Techniques

Exploring Effective Real-Time Ergonomic Guidance Methods for Immersive Virtual Reality Workspace

An Exploratory Study of Multimodal Interaction Modeling Based on Neural Computation

Evaluation of Hand-Based Interaction for Near-Field Mixed Reality with Optical See-Through Head-Mounted Displays

Toward an Efficient Hybrid Interaction Paradigm for Object Manipulation in Optical See-Through Mixed Reality

A human-in-the-loop haptic interaction with subjective evaluation

Research on Intelligent Experimental Equipment and Key Algorithms Based on Multimodal Fusion Perception

Subjective evaluation on human-machine-environment system of automotive driving simulation experiment platform

Optimizing Information Seeking for Multi-person Collaboration: Evaluating the Influence of Various Zoom Centers and Interaction Modals

On the Use of Multi-sensory Cues in Symmetric and Asymmetric Shared Collaborative Virtual Spaces

Perception and Decision-Making for Multi-Modal Interaction Based on Fuzzy Theory in the Dynamic Environment

Real-time multimodal interaction in virtual reality - a case study with a large virtual interface

A Multi-modal Virtual-Real Fusion System for Multi-task Human-Computer Interaction.

Research on multi-mode human-computer interaction design evaluation system based on virtual reality technology

Multimodal Coexistence Environment Design to Assist User Testing and Iterative Design of HiGame Emotional Interaction Design for Elderly

Optimization of Display Content Switching under Multiple Large-Screen Displays: A Multi-channel Interaction Usability Evaluation

Evaluation of Multimodal External Human–Machine Interface for Driverless Vehicles in Virtual Reality

Modality Capacity and Appropriateness in Multimodal Display of Complex Non-Semantic Information Stream