A Robust Mismatch Removal Method for Image Matching Based on the Fusion of the Local Features and the Depth

Xinpeng Ling,Jiahang Liu,Zexian Duan,Ji Luan

DOI: https://doi.org/10.3390/rs16111873

IF: 5

2024-05-25

Remote Sensing

Abstract:Feature point matching is a fundamental task in computer vision such as vision simultaneous localization and mapping (VSLAM) and structure from motion (SFM). Due to the similarity or interference of features, mismatches are often unavoidable. Therefore, how to eliminate mismatches is important for robust matching. Smoothness constraint is widely used to remove mismatch, but it cannot effectively deal with the issue in the rapidly changing scene. In this paper, a novel LCS-SSM (Local Cell Statistics and Structural Similarity Measurement) mismatch removal method is proposed. LCS-SSM integrates the motion consistency and structural similarity of a local image block as the statistical likelihood of matched key points. Then, the Random Sampling Consensus (RANSAC) algorithm is employed to preserve the isolated matches that do not satisfy the statistical likelihood. Experimental and comparative results on the public dataset show that the proposed LCS-SSM can effectively and reliably differentiate true and false matches compared with state-of-the-art methods, and can be used for robust matching in scenes with fast motion, blurs, and clustered noise.

environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively remove mismatches in image matching. Specifically, the paper proposes a new method - LCS - SSM (Local Cell Statistics and Structural Similarity Measurement) to improve the effect of removing mismatches in image matching based on local features and depth information. This method aims to solve the problem that traditional methods cannot effectively handle mismatches in rapidly changing scenes, especially in scenes with fast - moving objects, blurring and clustering noise.

### Specific problems solved by the paper:
1. **Mismatch problem**: During the image matching process, mismatches are inevitable due to feature similarity or interference. How to effectively remove these mismatches is the key to improving matching robustness.
2. **Matching problem in rapidly changing scenes**: Traditional smoothing - constraint methods perform poorly in rapidly changing scenes because these methods assume that the motion between pixels is consistent, but this assumption may not hold in edge regions.
3. **Processing of dynamic objects**: In scenes with dynamic objects, traditional mismatch - removal methods may not be able to effectively identify the correct matching points.

### Proposed methods:
- **LCS - SSM method**: Combines the motion consistency and structural similarity of local image patches and judges the correctness of matching points through statistical likelihood.
- **RANSAC algorithm**: Used to retain isolated correct matching points that do not satisfy statistical likelihood.

### Main contributions:
1. **Combination of smoothing constraint and structural similarity**: It not only depends on the smoothing constraint of texture features but also introduces the structural similarity of RGB images, which improves the robustness of matching.
2. **Use of depth images**: Utilizes the structural similarity in depth images to distinguish matching points on static and dynamic objects, solving the problem of the failure of the motion - consistency assumption in edge regions.
3. **Resampling method**: Retains isolated correct matching points through the RANSAC algorithm to ensure that these points are not misjudged as mismatches.

### Experimental results:
The paper conducted experiments on the TUM RGB - D dataset, using four different feature - point extraction algorithms: ORB, SIFT, SURF and AKAZE, and compared with three existing mismatch - removal methods: GMS, MAGSAC ++ and GC - RANSAC. The experimental results show that the LCS - SSM method is superior to other methods in terms of precision, recall and F1 - score, especially when dealing with fast - moving objects and complex scenes.

In conclusion, the paper proposes a novel and effective mismatch - removal method, which significantly improves the robustness and accuracy of image matching.