Semantic UV mapping to improve texture inpainting for indoor scenes

Jelle Vermandere,Maarten Bassier,Maarten Vergauwen

2024-07-12

Abstract:This work aims to improve texture inpainting after clutter removal in scanned indoor meshes. This is achieved with a new UV mapping pre-processing step which leverages semantic information of indoor scenes to more accurately match the UV islands with the 3D representation of distinct structural elements like walls and floors. Semantic UV Mapping enriches classic UV unwrapping algorithms by not only relying on geometric features but also visual features originating from the present texture. The segmentation improves the UV mapping and simultaneously simplifies the 3D geometric reconstruction of the scene after the removal of loose objects. Each segmented element can be reconstructed separately using the boundary conditions of the adjacent elements. Because this is performed as a pre-processing step, other specialized methods for geometric and texture reconstruction can be used in the future to improve the results even further.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper aims to solve the problem of how to repair textures after removing clutter objects from scanned indoor scenes. Specifically, the author proposes a new UV mapping pre - processing step. By leveraging the semantic information of indoor scenes, it more accurately matches the UV islands with the 3D representations of structural elements such as walls and floors, thereby improving the effect of texture repair.

### Main problems
1. **Hole repair after removing clutter objects**: In scanned indoor environments, there are usually some temporarily placed objects (such as furniture, decorations, etc.). These objects will cause occlusion and holes during the scanning process. After removing these objects, geometric and texture repairs are required for the missing parts.

2. **Limitations of existing texture repair methods**:
- Existing 2D texture repair methods are not effective in dealing with complex 3D geometric structures, especially when dealing with large - area occlusions or complex room geometric structures.
- Methods that directly predict colors in 3D space are limited by geometric resolution and it is difficult to generate fine details.

### Solutions
The author proposes a semantic - segmentation - based UV mapping method (Semantic UV Mapping). The main steps of this method include:
1. **Scene segmentation**: Use semantic instance segmentation techniques to separate loose objects and structural elements (such as walls, floors) in the scene and generate segmentation masks.
2. **Geometric reconstruction**: Remove loose objects according to the segmentation results and reconstruct the missing geometric parts one by one.
3. **Semantic UV mapping**: Create UV stitching lines based on the segmentation masks to ensure that the UV mapping better matches the 3D mesh, reduce distortion and maintain the connection of adjacent faces.
4. **Texture repair**: Perform texture repair on the newly generated geometric parts to ensure that the repaired texture is consistent with the original texture.

Through this method, the author can more effectively repair the holes left after removing objects and generate more realistic textures, which is especially suitable for simple geometric structures indoors.

### Summary
The core problem of this paper is to improve the texture repair quality of indoor scenes after removing clutter objects. By introducing semantic information to optimize UV mapping, it solves the shortcomings of existing methods in dealing with complex geometric structures.