Match Stereo Videos via Bidirectional Alignment

Junpeng Jing,Ye Mao,Anlan Qiu,Krystian Mikolajczyk

2024-09-30

Abstract:Video stereo matching is the task of estimating consistent disparity maps from rectified stereo videos. There is considerable scope for improvement in both datasets and methods within this area. Recent learning-based methods often focus on optimizing performance for independent stereo pairs, leading to temporal inconsistencies in videos. Existing video methods typically employ sliding window operation over time dimension, which can result in low-frequency oscillations corresponding to the window size. To address these challenges, we propose a bidirectional alignment mechanism for adjacent frames as a fundamental operation. Building on this, we introduce a novel video processing framework, BiDAStereo, and a plugin stabilizer network, BiDAStabilizer, compatible with general image-based methods. Regarding datasets, current synthetic object-based and indoor datasets are commonly used for training and benchmarking, with a lack of outdoor nature scenarios. To bridge this gap, we present a realistic synthetic dataset and benchmark focused on natural scenes, along with a real-world dataset captured by a stereo camera in diverse urban scenes for qualitative evaluation. Extensive experiments on in-domain, out-of-domain, and robustness evaluation demonstrate the contribution of our methods and datasets, showcasing improvements in prediction quality and achieving state-of-the-art results on various commonly used benchmarks. The project page, demos, code, and datasets are available at: \url{<a class="link-external link-https" href="https://tomtomtommi.github.io/BiDAVideo/" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address the consistency issues in video stereo matching, particularly temporal consistency in dynamic scenes. Specifically:

1. **Problems with Existing Methods**:
- Existing image-based methods typically optimize for independent stereo pairs, leading to temporal inconsistencies in videos.
- Existing video methods often use a sliding window operation to handle the temporal dimension, which may result in low-frequency oscillations.

2. **Proposed New Mechanism**:
- A Bidirectional Alignment mechanism is proposed to solve the feature matching problem between adjacent frames and is used as a fundamental operation.
- Based on this mechanism, a new video processing framework called BiDAStereo and a plugin stabilization network called BiDAStabilizer are introduced, the latter of which can be compatible with existing image-based methods.

3. **Dataset Improvements**:
- Current datasets mainly focus on indoor and synthetic objects, lacking datasets for outdoor natural scenes.
- To address this shortcoming, the paper proposes a synthetic dataset containing natural scenes and a diversified real-world dataset covering various urban environments.

Through these improvements, the paper demonstrates the superior performance of its method on multiple benchmarks and addresses the temporal consistency issues present in existing methods.