Abstract:Mixed reality systems are a promising direction of research that opens up great opportunities for interaction with virtual objects in the real world. Like any promising direction, mixed reality has a number of unresolved problems. One of these problems is the synthesis of natural lighting conditions for virtual objects, including the correct light interaction of virtual objects with the real world. Since virtual and real objects are located in different spaces, it is difficult to ensure their correct interaction. To create digital copies of real-world objects, machine learning tools and neural network technologies are employed. These methods are successfully used in computer vision for space orientation and environment reconstruction. As a solution, it is proposed to transfer all objects into the same information space: virtual space. This makes it possible to solve most of the problems associated with visual discomfort caused by the unnatural light interaction of real and virtual objects. Thus, the basic idea of the method is to recognize physical objects from point clouds and replace these objects with virtual CAD models. In other words, it implies semantic analysis of a scene and classification of objects with their subsequent transformation into polygonal models. In this study, we use competitive neural network architectures, which can provide state-of-the-art results. The test experiments are carried out on Semantic3D, ScanNet, and S3DIS, which are currently the largest datasets with point clouds that represent indoor scenes. For semantic segmentation and classification of 3D point clouds, we use the PointNeXt architecture based on PointNet, as well as modern methods of data augmentation in the process of learning. For geometry reconstruction, the Soft Rasterizer differentiable rendering method and the Total3Understanding neural network are considered.

Neural Rendering-Based Semantic Point Cloud Retrieval for Indoor Construction Progress Monitoring

Semantic Reconstruction based on RGB Image and Sparse Depth

Outdoor Scene 3D Reconstruction from Multiple Point Cloud

3D Reconstruction Approach for Outdoor Scene Based on Multiple Point Cloud Fusion

Research on Indoor 3D Reconstruction Technology Based on Semantic Visual Simultaneous Localization and Mapping

An 3D Point Cloud Reconstruction Approach from Single Image for Smart Power Plant

Reconstruction of High-Precision Semantic Map

3D Semantic Segmentation Using Deep Learning for Large-Scale Indoor Point Cloud

3D Reconstruction and Semantic Segmentation Method Combining PointNet and 3D-Lmnet from Single Image

Efficient Implicit Neural Reconstruction Using LiDAR

NeRF-Con : Neural Radiance Fields for Automated Construction Progress Monitoring

SEMANTIC SEGMENTATION OF INDOOR 3D POINT CLOUDS BY JOINT OPTIMIZATION OF GEOMETRIC FEATURES AND NEURAL NETWORKS

3D Scene Reconstruction with Sparse LiDAR Data and Monocular Image in Single Frame

Semantic Segmentation of Point Cloud With Novel Neural Radiation Field Convolution

SG-NeRF: Semantic-guided Point-based Neural Radiance Fields

3D Scene Reconstruction and Digitization Method for Mixed Reality Systems

A Specialized Pipeline for Efficient and Reliable 3D Semantic Model Reconstruction of Buildings from Indoor Point Clouds

Automated Semantic Segmentation of Indoor Point Clouds from Close-Range Images with Three-Dimensional Deep Learning

CONSTRUCTION OF A DUAL-TASK MODEL FOR INDOOR SCENE RECOGNITION AND SEMANTIC SEGMENTATION BASED ON POINT CLOUDS

Hybrid self-supervised learning-based architecture for construction progress monitoring

RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction