Markov Chain Monte Carlo: The Basics

Adrian Barbu,Song-Chun Zhu

DOI: https://doi.org/10.1007/978-981-13-2971-5_3

2020-01-01

Abstract:Imagine you enter a big national park (in the poem it is a mountain), your path is essentially a Markov chain in a bounded space. The frequency that you stay at an attraction spot is proportional to its popularity.

What problem does this paper attempt to address?

Markov Chain Monte Carlo (MCMC)

Vivek S. Borkar

DOI: https://doi.org/10.1007/s12045-022-1407-1

2022-07-23

Resonance

Abstract:This is a short account of the major highlights of Markov Chain Monte Carlo, with pointers to what lies beyond.
An effective introduction to the Markov Chain Monte Carlo method

Wenlong Wang

DOI: https://doi.org/10.48550/arXiv.2204.10145

2022-04-21

Computational Physics

Abstract:We present an intuitive, conceptual, but semi-rigorous introduction to the celebrated Markov Chain Monte Carlo method using a simple model of population dynamics as our motivation and focusing on a few elementary distributions. Conceptually, the population flow between cities closely resembles the random walk of a single walker in a state space. We start from two states, then three states, and finally the setup is fully generalized to many states of both discrete and continuous distributions. Despite the mathematical simplicity, the setup remarkably includes all the essential concepts of Markov Chain Monte Carlo without loss of generality, e.g., ergodicity, global balance and detailed balance, proposal or selection probability, acceptance probability, up to the underlying stochastic matrix, and error analysis. Our teaching experience suggests that most senior undergraduate students in physics can closely follow these materials without much difficulty.
An introduction to the Markov chain Monte Carlo method

Wenlong Wang

DOI: https://doi.org/10.1119/5.0122488

IF: 0.835

2022-12-02

American Journal of Physics

Abstract:We present an intuitive, conceptual, and semi-rigorous introduction to the Markov Chain Monte Carlo method using a simple model of population dynamics and focusing on a few elementary distributions. We start from two states, then three states, and finally generalize to many states with both discrete and continuous distributions. Despite the mathematical simplicity, our examples include the essential concepts of the Markov Chain Monte Carlo method, including ergodicity, global balance and detailed balance, proposal or selection probability, acceptance probability, the underlying stochastic matrix, and error analysis. Our experience suggests that most senior undergraduate students in physics can follow these materials without much difficulty.
The Invariance Principle for Some Class of Markov Chains

ZY Lin

DOI: https://doi.org/10.4213/tvp615

2000-01-01

Theory of Probability and Its Applications

Abstract:This work is devoted to the invariance principle for homogeneous Markov chains with a finite state space. The proof is based on a comparison of the asymptotic behavior of sums of random variables constructed by Markov chains with the same transition matrix and various initial distributions.
The importance Markov chain

Charly Andral,Randal Douc,Hugo Marival,Christian P. Robert

DOI: https://doi.org/10.1016/j.spa.2024.104316

IF: 1.43

2024-02-01

Stochastic Processes and their Applications

Abstract:The Importance Markov chain is a novel algorithm bridging the gap between rejection sampling and importance sampling, moving from one to the other through a tuning parameter. Based on a modified sample of an instrumental Markov chain targeting an instrumental distribution (typically via a MCMC kernel), the Importance Markov chain produces an extended Markov chain where the marginal distribution of the first component converges to the target distribution. For example, when targeting a multimodal distribution, the instrumental distribution can be chosen as a tempered version of the target which allows the algorithm to explore its modes more efficiently. We obtain a Law of Large Numbers and a Central Limit Theorem as well as geometric ergodicity for this extended kernel under mild assumptions on the instrumental kernel. Computationally, the algorithm is easy to implement and preexisting librairies can be used to sample from the instrumental distribution.

statistics & probability
Markov Chain Monte Carlo in Practice

Galin L. Jones,Qian Qin

DOI: https://doi.org/10.1146/annurev-statistics-040220-090158

IF: 7.9

2022-03-07

Annual Review of Statistics and Its Application

Abstract:Markov chain Monte Carlo (MCMC) is an essential set of tools for estimating features of probability distributions commonly encountered in modern applications. For MCMC simulation to produce reliable outcomes, it needs to generate observations representative of the target distribution, and it must be long enough so that the errors of Monte Carlo estimates are small. We review methods for assessing the reliability of the simulation effort, with an emphasis on those most useful in practically relevant settings. Both strengths and weaknesses of these methods are discussed. The methods are illustrated in several examples and in a detailed case study.

statistics & probability,mathematics, interdisciplinary applications
Markov chains: Structure and applications

R.L. Tweedie

DOI: https://doi.org/10.1016/s0169-7161(01)19025-5

2001-01-01

Handbook of Statistics

Abstract:Publisher The classical theory of Markov chains originally dealt only with chains on finite or countable spaces. In order to establish the fundamental aspects of Markov chain theory on more general spaces, the chapter presents an overview of results on countable spaces, including some recent and less familiar aspects even on these spaces. Although many real systems can be described in such a framework, many other applications require continuous or even more general spaces, and there has been a flourishing of work on non-countable spaces. This chapter integrates the general state space results with topological conditions, which guarantee that the obvious (but not always true) probabilistic properties are preserved when the space has topological features.
Markov Chain Monte Carlo Methods in Quantum Field Theories: A Modern Primer

Anosh Joseph

DOI: https://doi.org/10.48550/arXiv.1912.10997

2019-12-23

High Energy Physics - Theory

Abstract:We introduce and discuss Monte Carlo methods in quantum field theories. Methods of independent Monte Carlo, such as random sampling and importance sampling, and methods of dependent Monte Carlo, such as Metropolis sampling and Hamiltonian Monte Carlo, are introduced. We review the underlying theoretical foundations of Markov chain Monte Carlo. We provide several examples of Monte Carlo simulations, including one-dimensional simple harmonic oscillator, unitary matrix model exhibiting Gross-Witten-Wadia transition and a supersymmetric model exhibiting dynamical supersymmetry breaking.
Markov chains and mappings of distributions on compact spaces

David J. Aldous,Shi Feng

2024-04-01

Abstract:Consider a compact metric space $S$ and a pair $(j,k)$ with $k \ge 2$ and $1 \le j \le k$. For any probability distribution $\theta \in P(S)$, define a Markov chain on $S$ by: from state $s$, take $k$ i.i.d. ($\theta$) samples, and jump to the $j$'th closest. Such a chain converges in distribution to a unique stationary distribution, say $\pi_{j,k}(\theta)$. So this defines a mapping $\pi_{j,k}: P(S) \to P(S)$. What happens when we iterate this mapping? In particular, what are the fixed points of this mapping? We present a few rigorous results, to complement our extensive simulation study elsewhere.

Probability,Metric Geometry
A Markov Chain Approach to the Problem of Runs and Patterns

Zhiying Liang

1993-01-01

Abstract:A new approach. Converts the problem of runs and patterns into a problem of Markov chain with discrete state space.
The Markov chain asymptotics of random mapping graphs

Xinxing Chen,Jiangang Ying

DOI: https://doi.org/10.1016/j.anihpb.2006.05.004

2007-01-01

Annales de l'Institut Henri Poincare (B) Probability and Statistics

Abstract:In this paper the limit behavior of random mappings with n vertices is investigated. We first compute the asymptotic probability that a fixed class of finite non-intersected subsets of vertices are located in different components and use this result to construct a scheme of allocating particles with a related Markov chain. We then prove that the limit behavior of random mappings is actually embedded in such a scheme in a certain way. As an application, we shall give the asymptotic moments of the size of the largest component.
A Conceptual Introduction to Markov Chain Monte Carlo Methods

Joshua S. Speagle

DOI: https://doi.org/10.48550/arXiv.1909.12313

2020-03-08

Abstract:Markov Chain Monte Carlo (MCMC) methods have become a cornerstone of many modern scientific analyses by providing a straightforward approach to numerically estimate uncertainties in the parameters of a model using a sequence of random samples. This article provides a basic introduction to MCMC methods by establishing a strong conceptual understanding of what problems MCMC methods are trying to solve, why we want to use them, and how they work in theory and in practice. To develop these concepts, I outline the foundations of Bayesian inference, discuss how posterior distributions are used in practice, explore basic approaches to estimate posterior-based quantities, and derive their link to Monte Carlo sampling and MCMC. Using a simple toy problem, I then demonstrate how these concepts can be used to understand the benefits and drawbacks of various MCMC approaches. Exercises designed to highlight various concepts are also included throughout the article.

Other Statistics,Instrumentation and Methods for Astrophysics,Data Analysis, Statistics and Probability
A Tutorial on the Spectral Theory of Markov Chains

Eddie Seabrook,Laurenz Wiskott

DOI: https://doi.org/10.1162/neco_a_01611

2022-08-19

Abstract:Markov chains are a class of probabilistic models that have achieved widespread application in the quantitative sciences. This is in part due to their versatility, but is compounded by the ease with which they can be probed analytically. This tutorial provides an in-depth introduction to Markov chains, and explores their connection to graphs and random walks. We utilize tools from linear algebra and graph theory to describe the transition matrices of different types of Markov chains, with a particular focus on exploring properties of the eigenvalues and eigenvectors corresponding to these matrices. The results presented are relevant to a number of methods in machine learning and data mining, which we describe at various stages. Rather than being a novel academic study in its own right, this text presents a collection of known results, together with some new concepts. Moreover, the tutorial focuses on offering intuition to readers rather than formal understanding, and only assumes basic exposure to concepts from linear algebra and probability theory. It is therefore accessible to students and researchers from a wide variety of disciplines.

Machine Learning
Markov Chain Monte Carlo Methods, a survey with some frequent misunderstandings

Christian P. Robert,Wu Changye

DOI: https://doi.org/10.48550/arXiv.2001.06249

2020-01-17

Abstract:In this chapter, we review some of the most standard MCMC tools used in Bayesian computation, along with vignettes on standard misunderstandings of these approaches taken from Q \&~A's on the forum Cross-validated answered by the first author.

Computation
Markov chains and mappings of distributions on compact spaces II: Numerics and Conjectures

David J. Aldous,Madelyn Cruz,Shi Feng

2024-03-27

Abstract:Consider a compact metric space $S$ and a pair $(j,k)$ with $k \ge 2$ and $1 \le j \le k$. For any probability distribution $\theta \in P(S)$, define a Markov chain on $S$ by: from state $s$, take $k$ i.i.d. ($\theta$) samples, and jump to the $j$'th closest. Such a chain converges in distribution to a unique stationary distribution, say $\pi_{j,k}(\theta)$. This defines a mapping $\pi_{j,k}: P(S) \to P(S)$. What happens when we iterate this mapping? In particular, what are the fixed points of this mapping? A few results are proved in a companion article; this article, not intended for formal publication, records numerical studies and conjectures.

Probability
Geometry and Dynamics for Markov Chain Monte Carlo

Alessandro Barp,François-Xavier Briol,Anthony D. Kennedy,Mark Girolami

DOI: https://doi.org/10.1146/annurev-statistics-031017-100141

IF: 7.9

2018-03-07

Annual Review of Statistics and Its Application

Abstract:Markov chain Monte Carlo methods have revolutionized mathematical computation and enabled statistical inference within many previously intractable models. In this context, Hamiltonian dynamics have been proposed as an efficient way of building chains that can explore probability densities efficiently. The method emerges from physics and geometry, and these links have been extensively studied over the past thirty years. The aim of this review is to provide a comprehensive introduction to the geometric tools used in Hamiltonian Monte Carlo at a level accessible to statisticians, machine learners, and other users of the methodology with only a basic understanding of Monte Carlo methods. This will be complemented with some discussion of the most recent advances in the field, which we believe will become increasingly relevant to scientists.

statistics & probability,mathematics, interdisciplinary applications
Markov chain Monte Carlo without likelihoods

Paul Marjoram,John Molitor,Vincent Plagnol,Simon Tavaré

DOI: https://doi.org/10.1073/pnas.0306899100

IF: 11.1

2003-12-08

Proceedings of the National Academy of Sciences

Abstract:Many stochastic simulation approaches for generating observations from a posterior distribution depend on knowing a likelihood function. However, for many complex probability models, such likelihoods are either impossible or computationally prohibitive to obtain. Here we present a Markov chain Monte Carlo method for generating observations from a posterior distribution without the use of likelihoods. It can also be used in frequentist applications, in particular for maximum-likelihood estimation. The approach is illustrated by an example of ancestral inference in population genetics. A number of open problems are highlighted in the discussion.

multidisciplinary sciences
Markov Chains Approximations of jump-Diffusion Quantum Trajectories

Clement Pellegrini

DOI: https://doi.org/10.48550/arXiv.0803.2593

2008-03-18

Abstract:"Quantum trajectories" are solutions of stochastic differential equations also called Belavkin or Stochastic Schrödinger Equations. They describe random phenomena in quantum measurement theory. Two types of such equations are usually considered, one is driven by a one-dimensional Brownian motion and the other is driven by a counting process. In this article, we present a way to obtain more advanced models which use jump-diffusion stochastic differential equations. Such models come from solutions of martingale problems for infinitesimal generators. These generators are obtained from the limit of generators of classical Markov chains which describe discrete models of quantum trajectories. Furthermore, stochastic models of jump-diffusion equations are physically justified by proving that their solutions can be obtained as the limit of the discrete trajectories.

Probability
A Markov chain for metapopulations of small sizes with attraction landscape

Paulo C. Ventura,Eric K. Tokuda,Luciano da F. Costa,Francisco A. Rodrigues

DOI: https://doi.org/10.48550/arXiv.2111.13168

2022-10-18

Abstract:Mathematical models represent one of the fundamental ways of studying nature. In special, epidemic models have shown to be particularly useful in the understanding of the course of diseases and in the planning effective control policies. A particular type of epidemic model considers the individuals divided into populations. When studied in graphs, it is already known that the graph topology can play an important role in the evolution of the disease. At the same time, one may want to study the effect of the presence of an underlying \emph{attraction landscape} of the vertices, apart from the respectively underlying topology. In this work, we study metapopulations with small number of individuals in the presence of an attraction landscape. Individuals move across populations and get infected according to the SIS compartmental model. By using a Markov chain approach, we provide a numerical approximation to the prediction of the long-term prevalence of the disease. More specifically, an approach that combines two binomial distributions for mobility, with appropriate assumptions, is proposed to approximate the model. The problem setting is simulated through Monte-Carlo experiments and the obtained results are compared to the mathematic-analytical approach. Substantial agreement is observed between both approaches, which corroborates the effectiveness of the reported numerical approach. In addition, we also study the impact of different levels of attraction landscapes, as well as propagation on the local scale of the entire population. All in all, this study proposes a potentially effective approach to a mostly unexplored setting of disease transmission.

Physics and Society
Geometric allocation approaches in Markov chain Monte Carlo

S Todo,H Suwa

DOI: https://doi.org/10.1088/1742-6596/473/1/012013

2013-12-16

Journal of Physics: Conference Series

Abstract:The Markov chain Monte Carlo method is a versatile tool in statistical physics to evaluate multi-dimensional integrals numerically. For the method to work effectively, we must consider the following key issues: the choice of ensemble, the selection of candidate states, the optimization of transition kernel, algorithm for choosing a configuration according to the transition probabilities. We show that the unconventional approaches based on the geometric allocation of probabilities or weights can improve the dynamics and scaling of the Monte Carlo simulation in several aspects. Particularly, the approach using the irreversible kernel can reduce or sometimes completely eliminate the rejection of trial move in the Markov chain. We also discuss how the space-time interchange technique together with Walker's method of aliases can reduce the computational time especially for the case where the number of candidates is large, such as models with long-range interactions.

Markov Chain Monte Carlo: The Basics

Markov Chain Monte Carlo (MCMC)

An effective introduction to the Markov Chain Monte Carlo method

An introduction to the Markov chain Monte Carlo method

The Invariance Principle for Some Class of Markov Chains

The importance Markov chain

Markov Chain Monte Carlo in Practice

Markov chains: Structure and applications

Markov Chain Monte Carlo Methods in Quantum Field Theories: A Modern Primer

Markov chains and mappings of distributions on compact spaces

A Markov Chain Approach to the Problem of Runs and Patterns

The Markov chain asymptotics of random mapping graphs

A Conceptual Introduction to Markov Chain Monte Carlo Methods

A Tutorial on the Spectral Theory of Markov Chains

Markov Chain Monte Carlo Methods, a survey with some frequent misunderstandings

Markov chains and mappings of distributions on compact spaces II: Numerics and Conjectures

Geometry and Dynamics for Markov Chain Monte Carlo

Markov chain Monte Carlo without likelihoods

Markov Chains Approximations of jump-Diffusion Quantum Trajectories

A Markov chain for metapopulations of small sizes with attraction landscape

Geometric allocation approaches in Markov chain Monte Carlo