TSAR-MVS: Textureless-aware Segmentation and Correlative Refinement Guided Multi-View Stereo

Zhenlong Yuan,Jiakai Cao,Zhaoqi Wang,Zhaoxin Li

DOI: https://doi.org/10.1016/j.patcog.2024.110565

2024-08-30

Abstract:The reconstruction of textureless areas has long been a challenging problem in MVS due to lack of reliable pixel correspondences between images. In this paper, we propose the Textureless-aware Segmentation And Correlative Refinement guided Multi-View Stereo (TSAR-MVS), a novel method that effectively tackles challenges posed by textureless areas in 3D reconstruction through filtering, refinement and segmentation. First, we implement the joint hypothesis filtering, a technique that merges a confidence estimator with a disparity discontinuity detector to eliminate incorrect depth estimations. Second, to spread the pixels with confident depth, we introduce an iterative correlation refinement strategy that leverages RANSAC to generate 3D planes based on superpixels, succeeded by a weighted median filter for broadening the influence of accurately determined pixels. Finally, we present a textureless-aware segmentation method that leverages edge detection and line detection for accurately identify large textureless regions for further depth completion. Experiments on ETH3D, Tanks & Temples and Strecha datasets demonstrate the superior performance and strong generalization capability of our proposed method.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The paper aims to address the challenges of textureless regions in multi-view stereo (MVS) reconstruction. In MVS tasks, the reconstruction of textureless regions has always been a difficult problem due to the lack of reliable pixel correspondences. Although existing methods perform well in handling textured regions, they still have significant shortcomings when dealing with large textureless areas.

### Specific Problems

1. **Inaccurate depth estimation in textureless regions**: Most existing methods result in inaccurate depth estimation in textureless regions due to the lack of reliable photometric consistency.
2. **Noise and incompleteness in reconstruction results**: Errors in depth estimation in textureless regions lead to noise and incomplete triangulation in the reconstruction results.
3. **Insufficient fine reconstruction of detailed regions**: Existing methods often fail to preserve and refine detailed regions when handling textureless areas.

### Solution

To address these challenges, the authors propose a method called TSAR-MVS, which effectively solves the reconstruction problem of textureless regions through the following steps:

1. **Joint hypothesis filtering**: Combining confidence estimators and disparity discontinuity detectors to eliminate erroneous depth estimates.
2. **Iterative correlation refinement**: Using RANSAC to generate 3D planes and extending the influence of accurately determined pixels through weighted median filtering (WMF).
3. **Textureless-aware segmentation**: Combining edge detection and line detection techniques to accurately identify large textureless regions and further complete depth filling.

### Experimental Validation

Experiments were conducted on the ETH3D, Tanks & Temples, and Strecha datasets. The results show that TSAR-MVS significantly outperforms other MVS methods in reconstructing textureless regions and has strong generalization capabilities.

### Conclusion

TSAR-MVS effectively addresses the challenges of textureless regions in MVS reconstruction through innovative joint hypothesis filtering, iterative correlation refinement, and textureless-aware segmentation methods, improving the accuracy and completeness of the reconstruction results.