Synchronous mixed reality (SMR): A personalized virtual‐real fusion framework with high immersion and effective interaction

Jie Guo,Ting Ma,Dongdong Weng

DOI: https://doi.org/10.1002/jsid.1259

2023-10-13

Journal of the Society for Information Display

Abstract:The synchronous mixed reality (SMR) framework consists of three modules: The extraction module uses an instance segmentation algorithm to extract all physical information. The personalized information expression module creates virtual representations for physical objects and selectively expresses personalized information in VEs. The interaction module uses an algorithm to predict user interaction purpose and switches the form of interactive target. Virtual‐real fusion (VRF) technology plays a crucial role in the meta‐universe by bridging the gap between virtual environments (VEs) and physical environments (PEs). However, the current VRF system has the problem of single function and fixed integration, limiting the popularization and application of VRF technology. Therefore, this paper proposes a framework named synchronous mixed reality (SMR) to personalize the VRF system and maintain the balance between interaction efficiency and environmental immersion. We combine an instance segmentation algorithm with an interaction prediction algorithm to achieve this balance. To evaluate the effectiveness of the SMR framework, we design three scenarios based on the interaction properties of physical objects and measure environmental immersion, systematic interaction efficiency, and user experience. Our results demonstrate that the SMR framework meets diverse needs in various PEs while balancing immersion and interaction efficiency. Additionally, our framework significantly reduces collisions and negative emotions experienced by users in VEs. We anticipate that this framework will serve as a guide for constructing virtual‐real fusion systems in the future.

engineering, electrical & electronic,materials science, multidisciplinary,optics,physics, applied