Markov Random Fields in the Context of Stereo Vision

Lorenzo J.,Isabel Barbancho,Carlos Alberol

DOI: https://doi.org/10.5772/12953

2011-01-08

Abstract:Markov Random Fields in the Context of Stereo Vision | InTechOpen, Published on: 2011-01-08. Authors: Lorenzo J. Tardón, Isabel Barbancho and Carlos Alberola

What problem does this paper attempt to address?

Neural Markov Random Field for Stereo Matching

Tongfan Guan,Chen Wang,Yun-Hui Liu

2024-03-21

Abstract:Stereo matching is a core task for many computer vision and robotics applications. Despite their dominance in traditional stereo methods, the hand-crafted Markov Random Field (MRF) models lack sufficient modeling accuracy compared to end-to-end deep models. While deep learning representations have greatly improved the unary terms of the MRF models, the overall accuracy is still severely limited by the hand-crafted pairwise terms and message passing. To address these issues, we propose a neural MRF model, where both potential functions and message passing are designed using data-driven neural networks. Our fully data-driven model is built on the foundation of variational inference theory, to prevent convergence issues and retain stereo MRF's graph inductive bias. To make the inference tractable and scale well to high-resolution images, we also propose a Disparity Proposal Network (DPN) to adaptively prune the search space of disparity. The proposed approach ranks $1^{st}$ on both KITTI 2012 and 2015 leaderboards among all published methods while running faster than 100 ms. This approach significantly outperforms prior global methods, e.g., lowering D1 metric by more than 50% on KITTI 2015. In addition, our method exhibits strong cross-domain generalization and can recover sharp edges. The codes at

Computer Vision and Pattern Recognition
MRF-Based Stereo Correspondence and Virtual View Interpolation

Li Yao,DongXiao Li,Ming Zhang

DOI: https://doi.org/10.1109/IITA.2009.432

2009-01-01

Abstract:Stereo correspondence and virtual view synthesis play a crucial role in three-dimensional video systems. In this paper, the traditional stereo correspondence problem is modeled on Markov random fields (MRF) where the matching consistency is optimized by minimizing the global energy function. Both of the left-to-right disparity map and right-to-left disparity map are estimated simultaneously with the max-product belief propagation algorithm. The experimental results show that high quality intermediate views with significantly reduced hole regions can be obtained based on the bidirectional disparity maps.
Integration of stereo vision and optical flow using Markov random fields

S.P. Clifford,N.M. Nasrabadi,Clifford,Nasrabadi

DOI: https://doi.org/10.1109/icnn.1988.23893

1988-01-01

Abstract:A Markov random filed (MRF) model is used to derive the maximum a posteriori stereo-matched solution for an intensity matching and an edge-matching algorithm. The MRF-Gibbs distribution equivalence reduces the problem to finding appropriate energy functions that describe the constraints on the solution. If a simple MRF is used to model the data, the energy function will yield poor disparity results by smoothing across object boundaries. This problem is particularly apparent with occluded objects. The authors utilize edge and optical flow information to indicate object boundaries and improve the disparity solution in occluded regions. The similarity in intensity (or edge orientation) between corresponding sites of the left and right images is also used in the energy function. Although the authors use simulated annealing to find disparity solutions, a neural-network implementation with analog resistive networks can also be used, as shown by C. Kotch et. al. (1986).<>
Robust Pseudo Random Fields for Light-Field Stereo Matching

Chao-Tsung Huang

DOI: https://doi.org/10.1109/iccv.2017.11

2017-10-01

Abstract:Markov Random Fields are widely used to model light-field stereo matching problems. However, most previous approaches used fixed parameters and did not adapt to light-field statistics. Instead, they explored explicit vision cues to provide local adaptability and thus enhanced depth quality. But such additional assumptions could end up confining their applicability, e.g. algorithms designed for dense light fields are not suitable for sparse ones. In this paper, we develop an empirical Bayesian framework—Robust Pseudo Random Field—to explore intrinsic statistical cues for broad applicability. Based on pseudo-likelihood, it applies soft expectation-maximization (EM) for good model fitting and hard EM for robust depth estimation. We introduce novel pixel difference models to enable such adaptability and robustness simultaneously. We also devise an algorithm to employ this framework on dense, sparse, and even denoised light fields. Experimental results show that it estimates scene-dependent parameters robustly and converges quickly. In terms of depth accuracy and computation speed, it also outperforms state-of-the-art algorithms constantly.
Modeling stereopsis via Markov random field

Yansheng Ming,Zhanyi Hu

DOI: https://doi.org/10.1162/NECO_a_00005-Ming

Abstract:Markov random field (MRF) and belief propagation have given birth to stereo vision algorithms with top performance. This article explores their biological plausibility. First, an MRF model guided by physiological and psychophysical facts was designed. Typically an MRF-based stereo vision algorithm employs a likelihood function that reflects the local similarity of two regions and a potential function that models the continuity constraint. In our model, the likelihood function is constructed on the basis of the disparity energy model because complex cells are considered as front-end disparity encoders in the visual pathway. Our likelihood function is also relevant to several psychological findings. The potential function in our model is constrained by the psychological finding that the strength of the cooperative interaction minimizing relative disparity decreases as the separation between stimuli increases. Our model is tested on three kinds of stereo images. In simulations on images with repetitive patterns, we demonstrate that our model could account for the human depth percepts that were previously explained by the second-order mechanism. In simulations on random dot stereograms and natural scene images, we demonstrate that false matches introduced by the disparity energy model can be reliably removed using our model. A comparison with the coarse-to-fine model shows that our model is able to compute the absolute disparity of small objects with larger relative disparity. We also relate our model to several physiological findings. The hypothesized neurons of the model are selective for absolute disparity and have facilitative extra receptive field. There are plenty of such neurons in the visual cortex. In conclusion, we think that stereopsis can be implemented by neural networks resembling MRF.
Toward a Real-Time Belief Propagation Stereo Reconstruction for Computers, Robots, and Beyond

Minh Nguyen,Wei Qi Yan,Rui Gong,Patrice Delmas

DOI: https://doi.org/10.1109/ivcnz.2015.7761530

2015-01-01

Abstract:This paper introduces a fast and reliable Belief Propagation based stereo matching algorithm to be equipped with mobile real-time stereo vision applications; such as autonomous navigation of robots, vehicles, UAVs and drones, mobility aid for visually impaired, and more.. The algorithm approximates the minimum energy solution on graphical Markov Chains based on intensities of consecutive stereo image scanlines. It can achieve real-time performance (30 frames per seconds) while maintaining accurate results. This algorithm can also produce reliable results under non-ideal environments. It is obtained by estimating unknown factors to deal with brightness, color offsets, and vertical misalignments between two images. These factors are the ones to minimize the energy passed through the Markov chain model at the centers of stereo images.
Continuous Markov Random Fields for Robust Stereo Estimation

Koichiro Yamaguchi,Tamir Hazan,David McAllester,Raquel Urtasun

DOI: https://doi.org/10.48550/arXiv.1204.1393

2012-04-06

Computer Vision and Pattern Recognition

Abstract:In this paper we present a novel slanted-plane MRF model which reasons jointly about occlusion boundaries as well as depth. We formulate the problem as the one of inference in a hybrid MRF composed of both continuous (i.e., slanted 3D planes) and discrete (i.e., occlusion boundaries) random variables. This allows us to define potentials encoding the ownership of the pixels that compose the boundary between segments, as well as potentials encoding which junctions are physically possible. Our approach outperforms the state-of-the-art on Middlebury high resolution imagery as well as in the more challenging KITTI dataset, while being more efficient than existing slanted plane MRF-based methods, taking on average 2 minutes to perform inference on high resolution imagery.
Convolutional neural network based deep conditional random fields for stereo matching

Zhi Wang,Shiqiang Zhu,Yuehua Li,Zhengzhe Cui

DOI: https://doi.org/10.1016/j.jvcir.2016.08.022

IF: 2.887

2016-10-01

Journal of Visual Communication and Image Representation

Abstract:Stereo matching has been studied for many years and is still a challenge problem. The Markov Random Fields (MRF) model and the Conditional Random Fields (CRF) model based methods have achieved good performance recently. Based on these pioneer works, a deep conditional random fields based stereo matching algorithm is proposed in this paper, which draws a connection between the Convolutional Neural Network (CNN) and CRF. The object knowledge is used as a soft constraint, which can effectively improve the depth estimation accuracy. Moreover, we proposed a CNN potential function that learns the potentials of CRF in a CNN framework. The inference of the CRF model is formulated as a Recurrent Neural Network (RNN). A variety of experiments have been conducted on KITTI and Middlebury benchmark. The results show that the proposed algorithm can produce state-of-the-art results and outperform other MRF-based or CRF-based methods.

computer science, information systems, software engineering
Relaxing Symmetric Multiple Windows Stereo Using Markov Random Fields

Andrea Fusiello,Umberto Castellani,Vittorio Murino

DOI: https://doi.org/10.1007/3-540-44745-8_7

2001-01-01

Abstract:This paper introduces R-SMW, a new algorithm for stereo matching. The main aspect is the introduction of a Markov Random Field (MRF) model in the Symmetric Multiple Windows (SMW) stereo algorithm in order to obtain a non-deterministic relaxation. The SMW algorithm is an adaptive, multiple window scheme using left-right consistency to compute disparity. The MRF approach allows to combine in a single functional the disparity values coming from different windows, the left-right consistency constraint and regularization hypotheses. The optimal estimate of the disparity is obtained by minimizing an energy functional with simulated annealing. Results with both synthetic and real stereo pairs demonstrate the improvement over the original SMW algorithm, which was already proven to perform better than state-of-the-art algorithms.
A Temporally Streamlined Optimization Method for Stereo Video Correspondence

Li Yao -,DongXiao Li -,Ming Zhang -

DOI: https://doi.org/10.4156/ijact.vol4.issue2.29

2012-01-01

International Journal of Advancements in Computing Technology

Abstract:Multi-view plus associated depth information form a typical 3D video representation, in which the depth is usually a result of estimation procedure based on stereo correspondence. Traditional stereo correspondence methods deal with each stereo image pair separately. As a result, the depth sequence is temporally inconsistent. This paper presents a novel method to reconstruct spatio-temporally consistent depth maps. We formulate the stereo problem on 3D Markov random fields to exploit correlation in both spatial and temporal domain. Then a sequential belief propagation algorithm is applied for minimizing the global energy and temporal consistency is enforced during the process. In addition, our streamlined implementation is time and memory efficient. We perform quantitative evaluation as well as subject assessment on several test datasets, which shows that our method yields temporally consistent depth sequence and reduces flickering artifacts in the synthesized view while maintaining visual quality.
Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail

Luca Bartolomei,Fabio Tosi,Matteo Poggi,Stefano Mattoccia

2024-12-06

Abstract:We introduce Stereo Anywhere, a novel stereo-matching framework that combines geometric constraints with robust priors from monocular depth Vision Foundation Models (VFMs). By elegantly coupling these complementary worlds through a dual-branch architecture, we seamlessly integrate stereo matching with learned contextual cues. Following this design, our framework introduces novel cost volume fusion mechanisms that effectively handle critical challenges such as textureless regions, occlusions, and non-Lambertian surfaces. Through our novel optical illusion dataset, MonoTrap, and extensive evaluation across multiple benchmarks, we demonstrate that our synthetic-only trained model achieves state-of-the-art results in zero-shot generalization, significantly outperforming existing solutions while showing remarkable robustness to challenging cases such as mirrors and transparencies.

Computer Vision and Pattern Recognition
CNLPA-MVS: Coarse-Hypotheses Guided Non-Local PatchMatch Multi-View Stereo

Zhang Qitong,Luo Shan,Wang Lei,Feng Jieqing

DOI: https://doi.org/10.1007/s11390-021-1299-7

IF: 1.871

2021-01-01

Journal of Computer Science and Technology

Abstract:In multi-view stereo, unreliable matching in low-textured regions has a negative impact on the completeness of reconstructed models. Since the photometric consistency of low-textured regions is not discriminative under a local window, non-local information provided by the Markov Random Field (MRF) model can alleviate the matching ambiguity but is limited in continuous space with high computational complexity. Owing to its sampling and propagation strategy, PatchMatch multi-view stereo methods have advantages in terms of optimizing the continuous labeling problem. In this paper, we propose a novel method to address this problem, namely the Coarse-Hypotheses Guided Non-Local PatchMatch Multi-View Stereo (CNLPA-MVS), which takes the advantages of both MRF-based non-local methods and PatchMatch multi-view stereo and compensates for their defects mutually. First, we combine dynamic programing (DP) and sequential propagation along scanlines in parallel to perform CNLPA-MVS, thereby obtaining the optimal depth and normal hypotheses. Second, we introduce coarse inference within a universal window provided by winner-takes-all to eliminate the stripe artifacts caused by DP and improve completeness. Third, we add a local consistency strategy based on the hypotheses of similar color pixels sharing approximate values into CNLPA-MVS for further improving completeness. CNLPA-MVS was validated on public benchmarks and achieved state-of-the-art performance with high completeness.
RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching

Lahav Lipson,Zachary Teed,Jia Deng

DOI: https://doi.org/10.1109/3dv53792.2021.00032

2021-12-01

Abstract:We introduce RAFT-Stereo, a new deep architecture for rectified stereo based on the optical flow network RAFT [35]. We introduce multi-level convolutional GRUs, which more efficiently propagate information across the image. A modified version of RAFT-Stereo can perform accurate real-time inference. RAFT-stereo ranks first on the Middlebury leaderboard, outperforming the next best method on 1px error by 29% and outperforms all published work on the ETH3D two-view stereo benchmark. Code is available at https://github.com/princeton-vl/RAFT-Stereo.
Empirical Bayesian Light-Field Stereo Matching by Robust Pseudo Random Field Modeling

Chao-Tsung Huang

DOI: https://doi.org/10.1109/tpami.2018.2809502

IF: 23.6

2019-03-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Light-field stereo matching problems are commonly modeled by Markov Random Fields (MRFs) for statistical inference of depth maps. Nevertheless, most previous approaches did not adapt to image statistics but instead adopted fixed model parameters. They explored explicit vision cues, such as depth consistency and occlusion, to provide local adaptability and enhance depth quality. However, such additional assumptions could end up confining their applicability, e.g. algorithms designed for dense view sampling are not suitable for sparse one. In this paper, we get back to MRF fundamentals and develop an empirical Bayesian framework-Robust Pseudo Random Field-to explore intrinsic statistical cues for broad applicability. Based on pseudo-likelihoods with hidden soft-decision priors, we apply soft expectation-maximization (EM) for good model fitting and perform hard EM for robust depth estimation. We introduce novel pixel difference models to enable such adaptability and robustness simultaneously. Accordingly, we devise a stereo matching algorithm to employ this framework on dense, sparse, and even denoised light fields. It can be applied to both true-color and grey-scale pixels. Experimental results show that it estimates scene-dependent parameters robustly and converges quickly. In terms of depth accuracy and computation speed, it also outperforms state-of-the-art algorithms constantly.

computer science, artificial intelligence,engineering, electrical & electronic
MonStereo: When Monocular and Stereo Meet at the Tail of 3D Human Localization

Lorenzo Bertoni,Sven Kreiss,Taylor Mordan,Alexandre Alahi

DOI: https://doi.org/10.48550/arXiv.2008.10913

2021-03-23

Abstract:Monocular and stereo visions are cost-effective solutions for 3D human localization in the context of self-driving cars or social robots. However, they are usually developed independently and have their respective strengths and limitations. We propose a novel unified learning framework that leverages the strengths of both monocular and stereo cues for 3D human localization. Our method jointly (i) associates humans in left-right images, (ii) deals with occluded and distant cases in stereo settings by relying on the robustness of monocular cues, and (iii) tackles the intrinsic ambiguity of monocular perspective projection by exploiting prior knowledge of the human height distribution. We specifically evaluate outliers as well as challenging instances, such as occluded and far-away pedestrians, by analyzing the entire error distribution and by estimating calibrated confidence intervals. Finally, we critically review the official KITTI 3D metrics and propose a practical 3D localization metric tailored for humans.

Computer Vision and Pattern Recognition
Playing to Vision Foundation Model's Strengths in Stereo Matching

Chuang-Wei Liu,Qijun Chen,Rui Fan

2024-04-09

Abstract:Stereo matching has become a key technique for 3D environment perception in intelligent vehicles. For a considerable time, convolutional neural networks (CNNs) have remained the mainstream choice for feature extraction in this domain. Nonetheless, there is a growing consensus that the existing paradigm should evolve towards vision foundation models (VFM), particularly those developed based on vision Transformers (ViTs) and pre-trained through self-supervision on extensive, unlabeled datasets. While VFMs are adept at extracting informative, general-purpose visual features, specifically for dense prediction tasks, their performance often lacks in geometric vision tasks. This study serves as the first exploration of a viable approach for adapting VFMs to stereo matching. Our ViT adapter, referred to as ViTAS, is constructed upon three types of modules: spatial differentiation, patch attention fusion, and cross-attention. The first module initializes feature pyramids, while the latter two aggregate stereo and multi-scale contextual information into fine-grained features, respectively. ViTAStereo, which combines ViTAS with cost volume-based stereo matching back-end processes, achieves the top rank on the KITTI Stereo 2012 dataset and outperforms the second-best network StereoBase by approximately 7.9% in terms of the percentage of error pixels, with a tolerance of 3 pixels. Additional experiments across diverse scenarios further demonstrate its superior generalizability compared to all other state-of-the-art approaches. We believe this new paradigm will pave the way for the next generation of stereo matching networks.

Computer Vision and Pattern Recognition,Artificial Intelligence,Robotics
MVSCRF: Learning Multi-View Stereo with Conditional Random Fields

Youze Xue,Jiansheng Chen,Weitao Wan,Yiqing Huang,Cheng Yu,Tianpeng Li,Jiayu Bao

DOI: https://doi.org/10.1109/iccv.2019.00441

2019-01-01

Abstract:We present a deep-learning architecture for multi-view stereo with conditional random fields (MVSCRF). Given an arbitrary number of input images, we first use a U-shape neural network to extract deep features incorporating both global and local information, and then build a 3D cost volume for the reference camera. Unlike previous learning based methods, we explicitly constraint the smoothness of depth maps by using conditional random fields (CRFs) after the stage of cost volume regularization. The CRFs module is implemented as recurrent neural networks so that the whole pipeline can be trained end-to-end. Our results show that the proposed pipeline outperforms previous state-of-the-arts on large-scale DTU dataset. We also achieve comparable results with state-of-the-art learning based methods on outdoor Tanks and Temples dataset without fine-tuning, which demonstrates our method's generalization ability.
TemporalStereo: Efficient Spatial-Temporal Stereo Matching Network

Youmin Zhang,Matteo Poggi,Stefano Mattoccia

2023-08-03

Abstract:We present TemporalStereo, a coarse-to-fine stereo matching network that is highly efficient, and able to effectively exploit the past geometry and context information to boost matching accuracy. Our network leverages sparse cost volume and proves to be effective when a single stereo pair is given. However, its peculiar ability to use spatio-temporal information across stereo sequences allows TemporalStereo to alleviate problems such as occlusions and reflective regions while enjoying high efficiency also in this latter case. Notably, our model -- trained once with stereo videos -- can run in both single-pair and temporal modes seamlessly. Experiments show that our network relying on camera motion is robust even to dynamic objects when running on videos. We validate TemporalStereo through extensive experiments on synthetic (SceneFlow, TartanAir) and real (KITTI 2012, KITTI 2015) datasets. Our model achieves state-of-the-art performance on any of these datasets. Code is available at \url{<a class="link-external link-https" href="https://github.com/youmi-zym/TemporalStereo.git" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition

Markov Random Fields in the Context of Stereo Vision

Neural Markov Random Field for Stereo Matching

MRF-Based Stereo Correspondence and Virtual View Interpolation

Integration of stereo vision and optical flow using Markov random fields

Robust Pseudo Random Fields for Light-Field Stereo Matching

Modeling stereopsis via Markov random field

Toward a Real-Time Belief Propagation Stereo Reconstruction for Computers, Robots, and Beyond

Continuous Markov Random Fields for Robust Stereo Estimation

Convolutional neural network based deep conditional random fields for stereo matching

Relaxing Symmetric Multiple Windows Stereo Using Markov Random Fields

A Temporally Streamlined Optimization Method for Stereo Video Correspondence

Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail

CNLPA-MVS: Coarse-Hypotheses Guided Non-Local PatchMatch Multi-View Stereo

RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching

Empirical Bayesian Light-Field Stereo Matching by Robust Pseudo Random Field Modeling

MonStereo: When Monocular and Stereo Meet at the Tail of 3D Human Localization

Playing to Vision Foundation Model's Strengths in Stereo Matching

MVSCRF: Learning Multi-View Stereo with Conditional Random Fields

TemporalStereo: Efficient Spatial-Temporal Stereo Matching Network