Abstract:This paper is a survey of various proofs of the so called {\em fundamental theorem of Markov chains}: every ergodic Markov chain has a unique positive stationary distribution and the chain attains this distribution in the limit independent of the initial distribution the chain started with. As Markov chains are stochastic processes, it is natural to use probability based arguments for proofs. At the same time, the dynamics of a Markov chain is completely captured by its initial distribution, which is a vector, and its transition probability matrix. Therefore, arguments based on matrix analysis and linear algebra can also be used. The proofs discussed below use one or the other of these two types of arguments, except in one case where the argument is graph theoretic. Appropriate credits to the various proofs are given in the main text. Our first proof is entirely elementary, and yet the proof is also quite simple. The proof also suggests a mixing time bound, which we prove, but this bound in many cases will not be the best bound. One approach in proving the fundamental theorem breaks the proof in two parts: (i) show the existence of a unique positive stationary distribution for irreducible Markov chains, and (ii) assuming that an ergodic chain does have a stationary distribution, show that the chain will converge in the limit to that distribution irrespective of the initial distribution. For (i), we survey two proofs, one uses probability arguments, and the other uses graph theoretic arguments. For (ii), first we give a coupling based proof (coupling is a probability based technique), the other uses matrix analysis. Finally, we give a proof of the fundamental theorem using only linear algebra concepts.

A Note on the Bias and Kemeny's Constant in Markov Reward Processes with an Application to Markov Chain Perturbation

Kemeny's function for Markov chains and Markov renewal processes

The Invariance Principle for Some Class of Markov Chains

Bernstein-type Inequalities for Markov Chains and Markov Processes: A Simple and Robust Proof

On a boundary of the central limit theorem for strictly stationary, reversible Markov chains

On the Kemeny time for continuous-time reversible and irreversible Markov processes with applications to stochastic resetting and to conditioning towards forever-survival

The Arsenal of Perturbation Bounds for Finite Continuous-Time Markov Chains: A Perspective

Perturbations of Markov Chains

Remarks on limit theorems for reversible Markov processes

A note on entrywise perturbation theory for Markov chains

Risk-Sensitivity Vanishing Limit for Controlled Markov Processes

Kemeny's constant and the Lemoine point of a simplex

An invitation to adaptive Markov chain Monte Carlo convergence theory

Central limit theorem for bifurcating Markov chains under point-wise ergodic conditions

Markov Rewards Processes with Impulse Rewards and Absorbing States

Asymptotic properties for linear processes of functionals of reversible Markov Chains

Various proofs of the Fundamental Theorem of Markov Chains

Cutoff for mixtures of permuted Markov chains: reversible case

A Self-Normalized Central Limit Theorem for Markov Random Walks

On a limit behavior of a sequence of Markov processes perturbed in a neighborhood of a singular point

Universal Bounds on Information-Processing Capabilities of Markov Processes