Building a mixed reality system free from visual discomfort

Iuliia Golovchanskaia,Aleksandr Anikeev,Oleg Mirsky,Maksim Sorokin,Dmitry Zhdanov,Andrey Zhdanov,Igor Potemin
DOI: https://doi.org/10.1117/12.2597182
2021-09-12
Abstract:This research focuses on the possibility of building an alternative mixed reality (MR) system, which will eliminate all its main causes of visual discomfort and form a model of the real world in the virtual space, as much as possible corresponding to it. The relationship between virtual reality (VR) systems, which are limited to models of their own virtual world, and MR systems, which add virtual objects to the real world, is examined. This paper presents an approach based on generating a point cloud of static objects using RGB-D sensors of the MR device, its further classification and segmentation, and then searching for a similar CAD object in the appropriate database. The found virtual analogue, after appropriate transformations, replaces the real object of the scene. To create more realism, RGB images of real objects can be superimposed as textures on the corresponding virtual scene objects. The paper proposes a multimodal approach, which consists in searching for objects with similar modalities in databases. A virtual scene created in a single space using this approach eliminates the possibility of forming unnatural lighting and observation conditions for all objects, including virtual copies of real objects and added virtual objects.
What problem does this paper attempt to address?