Abstract:Stereo matching has become a key technique for 3D environment perception in intelligent vehicles. For a considerable time, convolutional neural networks (CNNs) have remained the mainstream choice for feature extraction in this domain. Nonetheless, there is a growing consensus that the existing paradigm should evolve towards vision foundation models (VFM), particularly those developed based on vision Transformers (ViTs) and pre-trained through self-supervision on extensive, unlabeled datasets. While VFMs are adept at extracting informative, general-purpose visual features, specifically for dense prediction tasks, their performance often lacks in geometric vision tasks. This study serves as the first exploration of a viable approach for adapting VFMs to stereo matching. Our ViT adapter, referred to as ViTAS, is constructed upon three types of modules: spatial differentiation, patch attention fusion, and cross-attention. The first module initializes feature pyramids, while the latter two aggregate stereo and multi-scale contextual information into fine-grained features, respectively. ViTAStereo, which combines ViTAS with cost volume-based stereo matching back-end processes, achieves the top rank on the KITTI Stereo 2012 dataset and outperforms the second-best network StereoBase by approximately 7.9% in terms of the percentage of error pixels, with a tolerance of 3 pixels. Additional experiments across diverse scenarios further demonstrate its superior generalizability compared to all other state-of-the-art approaches. We believe this new paradigm will pave the way for the next generation of stereo matching networks.

TMVM: Truncated Majority Voting Method to Discriminate and Reduce Mismatches for Local Matching Approaches

Performance evaluation of feature detection and matching in stereo visual odometry.

Variational Stereo Matching with Left Right Consistency Constraint

CNLPA-MVS: Coarse-Hypotheses Guided Non-Local PatchMatch Multi-View Stereo

Local Stereo Matching with Improved Matching Cost and Disparity Refinement.

Ensemble Learning with Advanced Fast Image Filtering Features for Semi-Global Matching

Mutual information based non-parametric transform stereo matching algorithm

A Fast Local Stereo Matching Insensitive to Illumination Conditions

Local Expansion Moves for Stereo Matching Based on Random Sample Consensus Confidence

MRF-Based Stereo Correspondence and Virtual View Interpolation

Stereo Matching with Improved Radiometric Invariant Matching Cost and Disparity Refinement

Stereo Matching Using Neighboring System Constructed with MST

The Study of Stereo Matching Optimization Based on Multi-Baseline Trinocular Model

Effective local stereo matching by extended triangular interpolation

Local PatchMatch Based on Superpixel Cut for Efficient High-resolution Stereo Matching

Segment-Based Disparity Refinement With Occlusion Handling for Stereo Matching

Stereo Matching with Space-Constrained Cost Aggregation and Segmentation-Based Disparity Refinement

Stereo matching method based on high-precision occlusion-recovering and discontinuity-preserving

Insights into local stereo matching: Evaluation of disparity refinement approaches

Topologically Clustering: a Method for Discarding Mismatches

Playing to Vision Foundation Model's Strengths in Stereo Matching