Dream-Experiment: A MR User Interface with Natural Multi-channel Interaction for Virtual Experiments

Tianren Luo,Mingmin Zhang,Zhigeng Pan,Zheng Li,Ning Cai,Jinda Miao,Youbin Chen,Mingxi Xu

DOI: https://doi.org/10.1109/tvcg.2020.3023602

IF: 5.2

2020-12-01

IEEE Transactions on Visualization and Computer Graphics

Abstract:This paper studies a set of MR technologies for middle school experimental teaching environments and develops a multi-channel MR user interface called Dream-Experiment. The goal of Dream-Experiment is to improve the traditional MR user interface, so that users can get a real, natural 3D interactive experience like real experiments, but without danger and pollution. In terms of visual presentation, we design multi-camera collaborative registration to realize robust 6-DoF MR interactive space, and also define a complete rendering pipeline to provide improved processing of virtual-real objects' occlusion including translucent devices. In the virtual-real interaction, we provide six interaction modes that support visual interaction, tangible interaction, virtual-real gestures with touching, voice, thermal feeling, and olfactory feeling. After users' testing, we find that Dream-Experiment has better interactive efficiency and user experience than traditional MR environments.

computer science, software engineering

What problem does this paper attempt to address?

The paper aims to address the issues of Mixed Reality (MR) technology in secondary school experimental teaching environments. Specifically, real experiments have many limitations, such as danger (flames, explosions, corrosion), pollution, high cost (equipment and chemicals), and difficulty in understanding (invisible phenomena like magnetic fields, charges, currents). Existing AR/MR systems often lack correct occlusion relationships in visual presentation, which can interfere with users' spatial cognition and reduce the system's practicality. Moreover, most current MR experimental applications mainly rely on specific gesture actions or handheld markers for interaction, which cannot provide the same operational feel and tactile experience as real experiments.

To solve these problems, researchers designed and developed a multi-channel MR user interface called Dream-Experiment. The goal of this system is to improve traditional MR user interfaces, allowing users to have a natural 3D interactive experience similar to real experiments while avoiding dangers and pollution. The specific contributions include:

1. **Multi-camera collaborative registration**: Developed a multi-camera collaborative registration algorithm to create a robust 6 degrees of freedom (6-DoF) interactive space.
2. **Virtual-real occlusion method**: Proposed a multi-layer, multi-material, pixel-level accurate virtual-real occlusion rendering algorithm, including opaque and semi-transparent objects.
3. **Multimodal interaction**: Simulated the multi-sensory experience of real experiments and designed a user interface that supports six interaction modes: visual interaction, physical interaction, touch-based virtual-real gesture interaction, voice, temperature, and smell.

With these improvements, Dream-Experiment can provide better rendering effects and interactive experiences in complex environments.