Abstract:We study continuous-time Markov chains on the non-negative integers under mild regularity conditions (in particular, the set of jump vectors is finite and both forward and backward jumps are possible). Based on the so-called flux balance equation, we derive an iterative formula for calculating stationary measures. Specifically, a stationary measure $\pi(x)$ evaluated at $x\in\mathbb{N}_0$ is represented as a linear combination of a few generating terms, similarly to the characterization of a stationary measure of a birth-death process, where there is only one generating term, $\pi(0)$. The coefficients of the linear combination are recursively determined in terms of the transition rates of the Markov chain. For the class of Markov chains we consider, there is always at least one stationary measure (up to a scaling constant). We give various results pertaining to uniqueness and non-uniqueness of stationary measures, and show that the dimension of the linear space of signed invariant measures is equal to the number of generating terms. A minimization problem is constructed in order to compute stationary measures numerically. Moreover, a heuristic linear approximation scheme is suggested for the same purpose by first approximating the generating terms. The correctness of the linear approximation scheme is justified in some special cases. Furthermore, a decomposition of the state space into different types of states (open and closed irreducible classes, and trapping, escaping and neutral states) is presented. Applications to stochastic reaction networks are well illustrated.

Using Board Games and Mathematica to Teach the Fundamentals of Finite Stationary Markov Chains

Teaching Risk and Probability: Building the Monopoly® Board Game Into a Probability Simulator

An introduction to the Markov chain Monte Carlo method

An effective introduction to the Markov Chain Monte Carlo method

Elementary submodels, coding strategies, and an infinite real number game

Using Symbolic Computation to analyze some Children's Board Games

Markov chains and mappings of distributions on compact spaces

Statistical Methods in Markov Chains

Multi-strategy evolutionary games: A Markov chain approach

Markov chains and mappings of distributions on compact spaces II: Numerics and Conjectures

Markov chains: Structure and applications

Stationary measures of continuous time Markov chains with applications to stochastic reaction networks

Analytic-geometric methods for finite Markov chains with applications to quasi-stationarity

A Tutorial on the Spectral Theory of Markov Chains

Accurately Computing Expected Visiting Times and Stationary Distributions in Markov Chains

Algorithms for Markov Binomial Chains

Interacting Markov chain Monte Carlo methods for solving nonlinear measure-valued equations

Reconciling Rationality and Stochasticity: Rich Behavioral Models in Two-Player Games

Probabilistic Model Checking for Continuous Time Markov Chains via Sequential Bayesian Inference

An Algorithm for Computing Stochastically Stable Distributions with Applications to Multiagent Learning in Repeated Games

Modeling Replicator Dynamics in Stochastic Games Using Markov Chain Method.