MP-MVS: Multi-Scale Windows PatchMatch and Planar Prior Multi-View Stereo

Rongxuan Tan,Qing Wang,Xueyan Wang,Chao Yan,Yang Sun,Youyang Feng

2023-09-23

Abstract:Significant strides have been made in enhancing the accuracy of Multi-View Stereo (MVS)-based 3D reconstruction. However, untextured areas with unstable photometric consistency often remain incompletely reconstructed. In this paper, we propose a resilient and effective multi-view stereo approach (MP-MVS). We design a multi-scale windows PatchMatch (mPM) to obtain reliable depth of untextured areas. In contrast with other multi-scale approaches, which is faster and can be easily extended to PatchMatch-based MVS approaches. Subsequently, we improve the existing checkerboard sampling schemes by limiting our sampling to distant regions, which can effectively improve the efficiency of spatial propagation while mitigating outlier generation. Finally, we introduce and improve planar prior assisted PatchMatch of ACMP. Instead of relying on photometric consistency, we utilize geometric consistency information between multi-views to select reliable triangulated vertices. This strategy can obtain a more accurate planar prior model to rectify photometric consistency measurements. Our approach has been tested on the ETH3D High-res multi-view benchmark with several state-of-the-art approaches. The results demonstrate that our approach can reach the state-of-the-art. The associated codes will be accessible at <a class="link-external link-https" href="https://github.com/RongxuanTan/MP-MVS" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the issue of inaccurate depth estimation in textureless regions in multi-view stereo (MVS) reconstruction. Specifically, current PatchMatch-based methods struggle with depth estimation in textureless regions due to the lack of sufficient texture information, which in turn affects the completeness of scene reconstruction. This paper proposes a multi-scale window PatchMatch (mPM) method, combined with a plane prior-assisted PatchMatch technique, aiming to improve the accuracy of depth estimation in textureless regions and the completeness of scene reconstruction.

### Main Contributions
1. **Multi-Scale Window PatchMatch (mPM)**:
- Proposes an efficient multi-scale window PatchMatch method that can use multi-scale patches for photometric consistency calculation in a single PatchMatch process.
- Compared to other multi-scale methods, mPM is faster and easier to extend to PatchMatch-based MVS methods.

2. **Improved Checkerboard Sampling Scheme**:
- By limiting the distance of sampling points and sampling the optimal plane hypothesis only from eight farther neighborhoods, it effectively suppresses the generation of outliers and improves the efficiency and reliability of spatial propagation.

3. **Geometry Consistency-Assisted Plane Prior Model**:
- Utilizes geometric consistency information between multiple views to select reliable triangulation vertices instead of relying on photometric consistency, thereby generating a more accurate plane prior model and improving the completeness of textureless scene reconstruction.

### Experimental Results
- Experiments were conducted on the ETH3D high-resolution multi-view benchmark dataset, and the results show that the proposed method achieves state-of-the-art performance.
- Compared to existing SOTA methods, the proposed method performs excellently in both indoor and outdoor scenes, especially in terms of depth estimation in textureless regions and the completeness of scene reconstruction.

### Conclusion
The proposed method effectively addresses the issue of inaccurate depth estimation in textureless regions through multi-scale window PatchMatch, an improved checkerboard sampling scheme, and a geometry consistency-assisted plane prior model, significantly improving the accuracy and completeness of multi-view stereo reconstruction.