Circulations of Markov-Chains with Continuous-Time and the Probability Interpretation of Some Determinants

MP QIAN,M QIAN,C QIAN

1984-01-01

Abstract:In this paper, the relation between the irreversibility and circulation of a Markov chainwith continuous time parameter is considered. By the calculation of determinants, it showshow the probability approach in[1] is related to T. Hill's results.

What problem does this paper attempt to address?

The Invariance Principle for Some Class of Markov Chains

ZY Lin

DOI: https://doi.org/10.4213/tvp615

2000-01-01

Theory of Probability and Its Applications

Abstract:This work is devoted to the invariance principle for homogeneous Markov chains with a finite state space. The proof is based on a comparison of the asymptotic behavior of sums of random variables constructed by Markov chains with the same transition matrix and various initial distributions.
Higher Order Ergodic Markov Chains and First Passage Times

Lixing Han,Kelun Wang,Jianhong Xu

DOI: https://doi.org/10.1080/03081087.2021.1968782

2022-01-01

Abstract:We study the relationship between ergodicity and irreducibility of transition probability tensors. We then introduce the notions of first passage times and mean first passage times, both in the tensor form, for higher order Markov chains and prove some properties of these quantities. Numerical examples are also given to illustrate our results.
2 Circulation Distribution, Entropy Production and Irreversibility of Finite Markov Chains with Continuous Parameter

Da-Quan Jiang,Min Qian,Min-Ping Qian

DOI: https://doi.org/10.1007/978-3-540-40957-1_3

2004-01-01

Abstract: • 2.1 Circulation Distribution • 2.2 Irreversibility and Entropy Production • 2.3 Large Deviations and Fluctuation Theorem
Circulation Distribution, Entropy Production And Irreversibility Of Denumerable Markov Chains

Da-Quan Jiang,Min Qian,Min-Ping Qian

DOI: https://doi.org/10.1007/978-3-540-40957-1_2

2004-01-01

Abstract: • 1.1 Directed Circuits, Cycles and Passage Functions • 1.2 The Derived Chain • 1.3 Circulation Distribution of Recurrent Markov Chains • 1.4 Irreversibility and Entropy Production • 1.5 Large Deviations and Fluctuation Theorem • 1.6 Appendix
Markov Chains On Continuous State Space

Quan-Lin Li

DOI: https://doi.org/10.1007/978-3-642-11492-2_5

2010-01-01

Abstract:In this chapter, we discuss Markov chains on continuous state space. We first analyze a discrete-time Markov chain on continuous state space, and then discuss a discrete-time Markov chain on a bivariate state space. Applying the censoring technique, we provide expression for the RG-factorizations, which are used to derive the stationary probability of the Markov chain. Further, we consider a continuous-time Markov chain on continuous state space. Specifically, we deal with a continuous-time level-dependent QBD process with continuous phase variable, and provide orthonormal representations for the R-, U- and G-measures, which lead to the matrix-structured computation of the stationary probability. As an application, we introduce continuous-phase type (CPH) distribution and continuous-phase Markovian arrival process (CMAP), and then analyze a CMAP/CPH/1 queue. Finally, we study a piecewise deterministic Markov process, which is applied to deal with more general queues such as the GI/G/c queue.
Reversibility and Entropy Production of Inhomogeneous Markov Chains

Hao Ge,Da-Quan Jiang,Min Qian

DOI: https://doi.org/10.1239/jap/1165505205

2006-01-01

Abstract:In this paper we introduce the concepts of instantaneous reversibility and instantaneous entropy production rate for inhomogeneous Markov chains with denumerable state spaces. The following statements are proved to be equivalent: the inhomogeneous Markov chain is instantaneously reversible; it is in detailed balance; its entropy production rate vanishes. In particular, for a time-periodic birth-death chain, which can be regarded as a simple version of a physical model (Brownian motors), we prove that its rotation number is 0 when it is instantaneously reversible or periodically reversible. Hence, in our model of Markov chains, the directed transport phenomenon of Brownian motors can occur only in nonequilibrium and irreversible systems.
On Three Classical Problems for Markov Chains with Continuous Time Parameters

MF CHEN

DOI: https://doi.org/10.2307/3214868

1991-01-01

Abstract:For a given transition rate, i.e., aQ-matrixQ= (qij) on a countable state space, the uniqueness of theQ-semigroupP(t) = (Pij(t)),the recurrence and the positive recurrence of the corresponding Markov chain are three fundamental and classical problems, treated in many textbooks. As an addition, this paper introduces some practical results motivated from the study of a type of interacting particle systems, reaction diffusion processes. The main results are theorems (1.11), (1.17) and (1.18). Their proofs are quite straightforward.
Explosion, implosion, and moments of passage times for continuous-time Markov chains: a semimartingale approach

Mikhail Menshikov,Dimitri Petritis

DOI: https://doi.org/10.1016/j.spa.2014.03.001

2012-06-14

Abstract:We establish general theorems quantifying the notion of recurrence --- through an estimation of the moments of passage times --- for irreducible continuous-time Markov chains on countably infinite state spaces. Sharp conditions of occurrence of the phenomenon of explosion are also obtained. A new phenomenon of implosion is introduced and sharp conditions for its occurrence are proven. The general results are illustrated by treating models having a difficult behaviour even in discrete time.

Probability,Mathematical Physics
Decisive Markov Chains

Parosh Aziz Abdulla,Noomene Ben Henda,Richard Mayr

DOI: https://doi.org/10.2168/LMCS-3%284%3A7%292007

2007-11-09

Abstract:We consider qualitative and quantitative verification problems for infinite-state Markov chains. We call a Markov chain decisive w.r.t. a given set of target states F if it almost certainly eventually reaches either F or a state from which F can no longer be reached. While all finite Markov chains are trivially decisive (for every set F), this also holds for many classes of infinite Markov chains. Infinite Markov chains which contain a finite attractor are decisive w.r.t. every set F. In particular, this holds for probabilistic lossy channel systems (PLCS). Furthermore, all globally coarse Markov chains are decisive. This class includes probabilistic vector addition systems (PVASS) and probabilistic noisy Turing machines (PNTM). We consider both safety and liveness problems for decisive Markov chains, i.e., the probabilities that a given set of states F is eventually reached or reached infinitely often, respectively. 1. We express the qualitative problems in abstract terms for decisive Markov chains, and show an almost complete picture of its decidability for PLCS, PVASS and PNTM. 2. We also show that the path enumeration algorithm of Iyer and Narasimha terminates for decisive Markov chains and can thus be used to solve the approximate quantitative safety problem. A modified variant of this algorithm solves the approximate quantitative liveness problem. 3. Finally, we show that the exact probability of (repeatedly) reaching F cannot be effectively expressed (in a uniform way) in Tarski-algebra for either PLCS, PVASS or (P)NTM.

Logic in Computer Science,Discrete Mathematics
Hoeffding's inequality for continuous-time Markov chains

Jinpeng Liu,Yuanyuan Liu,Lin Zhou

2024-04-23

Abstract:Hoeffding's inequality is a fundamental tool widely applied in probability theory, statistics, and machine learning. In this paper, we establish Hoeffding's inequalities specifically tailored for an irreducible and positive recurrent continuous-time Markov chain (CTMC) on a countable state space with the invariant probability distribution ${\pi}$ and an $\mathcal{L}^{2}(\pi)$-spectral gap ${\lambda}(Q)$. More precisely, for a function $g:E\to [a,b]$ with a mean $\pi(g)$, and given $t,\varepsilon>0$, we derive the inequality \[ \mathbb{P}_{\pi}\left(\frac{1}{t} \int_{0}^{t} g\left(X_{s}\right)\mathrm{d}s-\pi (g) \geq \varepsilon \right) \leq \exp\left\{-\frac{{\lambda}(Q)t\varepsilon^2}{(b-a)^2} \right\}, \] which can be viewed as a generalization of Hoeffding's inequality for discrete-time Markov chains (DTMCs) presented in [J. Fan et al., J. Mach. Learn. Res., 22(2022), pp. 6185-6219] to the realm of CTMCs. The key analysis enabling the attainment of this inequality lies in the utilization of the techniques of skeleton chains and augmented truncation approximations. Furthermore, we also discuss Hoeffding's inequality for a jump process on a general state space.

Probability,Statistics Theory
The Derivation of Markov Chain Properties using Generalized Matrix Inverses

Jeffrey J. Hunter

DOI: https://doi.org/10.48550/arXiv.1112.3404

2011-12-15

Abstract:The analysis of many problems of interest associated with Markov chains, e.g. stationary distributions, moments of first passage time distributions and moments of occupation time random variables, involves the solution of a system of linear equations involving I - P, where P is the transition matrix of a finite, irreducible, discrete time Markov chain. Generalized inverses play an important role in the solution of such singular sets of equations. In this presentation we survey the application of generalized matrix inverses to the aforementioned problems.

Probability
Markov chain properties in terms of column sums of the transition matrix

Jeffrey J. Hunter

DOI: https://doi.org/10.48550/arXiv.1111.0152

2011-11-01

Abstract:Questions are posed regarding the influence that the column sums of the transition probabilities of a stochastic matrix (with row sums all one) have on the stationary distribution, the mean first passage times and the Kemeny constant of the associated irreducible discrete time Markov chain. Some new relationships, including some inequalities, and partial answers to the questions, are given using a special generalized matrix inverse that has not previously been considered in the literature on Markov chains.

Probability
Stability Of Circulation Decompositions And Self-Organization Phenomena

Sj Yan,Mf Chen

DOI: https://doi.org/10.1016/s0252-9602(18)30638-6

1984-01-01

Acta Mathematica Scientia

Abstract:The circulation decomposition for a Markov chain with discrete parameteir was introduced in 1979 by Qian Min-ping. However, from physical and mathematical point of view, the analogue for a Markov chain with continuous parameter may be more interesting.
Markov Chains Approximations of jump-Diffusion Quantum Trajectories

Clement Pellegrini

DOI: https://doi.org/10.48550/arXiv.0803.2593

2008-03-18

Abstract:"Quantum trajectories" are solutions of stochastic differential equations also called Belavkin or Stochastic Schrödinger Equations. They describe random phenomena in quantum measurement theory. Two types of such equations are usually considered, one is driven by a one-dimensional Brownian motion and the other is driven by a counting process. In this article, we present a way to obtain more advanced models which use jump-diffusion stochastic differential equations. Such models come from solutions of martingale problems for infinitesimal generators. These generators are obtained from the limit of generators of classical Markov chains which describe discrete models of quantum trajectories. Furthermore, stochastic models of jump-diffusion equations are physically justified by proving that their solutions can be obtained as the limit of the discrete trajectories.

Probability
Markov chains and mappings of distributions on compact spaces II: Numerics and Conjectures

David J. Aldous,Madelyn Cruz,Shi Feng

2024-03-27

Abstract:Consider a compact metric space $S$ and a pair $(j,k)$ with $k \ge 2$ and $1 \le j \le k$. For any probability distribution $\theta \in P(S)$, define a Markov chain on $S$ by: from state $s$, take $k$ i.i.d. ($\theta$) samples, and jump to the $j$'th closest. Such a chain converges in distribution to a unique stationary distribution, say $\pi_{j,k}(\theta)$. This defines a mapping $\pi_{j,k}: P(S) \to P(S)$. What happens when we iterate this mapping? In particular, what are the fixed points of this mapping? A few results are proved in a companion article; this article, not intended for formal publication, records numerical studies and conjectures.

Probability
One-dimensional Continuous-Time Quantum Markov Chains: qubit probabilities and measures

Manuel D. De la Iglesia,Carlos F. Lardizabal

2024-02-25

Abstract:Quantum Markov chains (QMCs) are positive maps on a trace-class space describing open quantum dynamics on graphs. Such objects have a statistical resemblance with classical random walks, while at the same time it allows for internal (quantum) degrees of freedom. In this work we study continuous-time QMCs on the integer line, half-line and finite segments, so that we are able to obtain exact probability calculations in terms of the associated matrix-valued orthogonal polynomials and measures. The methods employed here are applicable to a wide range of settings, but we will restrict to classes of examples for which the Lindblad generators are induced by a single positive map, and such that the Stieltjes transforms of the measures and their inverses can be calculated explicitly.

Quantum Physics,Mathematical Physics,Classical Analysis and ODEs
Statistical Methods in Markov Chains

Patrick Billingsley

DOI: https://doi.org/10.1214/aoms/1177705136

1961-03-01

The Annals of Mathematical Statistics

Abstract:This paper is an expository survey of the mathematical aspects of statistical inference as it applies to finite Markov chains, the problem being to draw inferences about the transition probabilities from one long, unbroken observation $\{x_1, x_2, \cdots, x_n\}$ on the chain. The topics covered include Whittle's formula, chi-square and maximum-likelihood methods, estimation of parameters, and multiple Markov chains. At the end of the paper it is briefly indicated how these methods can be applied to a process with an arbitrary state space or a continuous time parameter. Section 2 contains a simple proof of Whittle's formula; Section 3 provides an elementary and self-contained development of the limit theory required for the application of chi-square methods to finite chains. In the remainder of the paper, the results are accompanied by references to the literature, rather than by complete proofs. As is usual in a review paper, the emphasis reflects the author's interests. Other general accounts of statistical inference on Markov processes will be found in Grenander [53], Bartlett [9] and [10], Fortet [35], and in my monograph [18]. I would like to thank Paul Meier for a number of very helpful discussions on the topics treated in this paper, particularly those of Section 3.
Stationary Probability Vectors of Higher-order Markov Chains

Chi-Kwong Li,Shixiao Zhang

DOI: https://doi.org/10.48550/arXiv.1303.3639

2014-02-23

Abstract:We consider the higher-order Markov Chain, and characterize the second order Markov chains admitting every probability distribution vector as a stationary vector. The result is used to construct Markov chains of higher-order with the same property. We also study conditions under which the set of stationary vectors of the Markov chain has a certain affine dimension.

Probability
Markov chains and mappings of distributions on compact spaces

David J. Aldous,Shi Feng

2024-04-01

Abstract:Consider a compact metric space $S$ and a pair $(j,k)$ with $k \ge 2$ and $1 \le j \le k$. For any probability distribution $\theta \in P(S)$, define a Markov chain on $S$ by: from state $s$, take $k$ i.i.d. ($\theta$) samples, and jump to the $j$'th closest. Such a chain converges in distribution to a unique stationary distribution, say $\pi_{j,k}(\theta)$. So this defines a mapping $\pi_{j,k}: P(S) \to P(S)$. What happens when we iterate this mapping? In particular, what are the fixed points of this mapping? We present a few rigorous results, to complement our extensive simulation study elsewhere.

Probability,Metric Geometry
Markov chains: Structure and applications

R.L. Tweedie

DOI: https://doi.org/10.1016/s0169-7161(01)19025-5

2001-01-01

Handbook of Statistics

Abstract:Publisher The classical theory of Markov chains originally dealt only with chains on finite or countable spaces. In order to establish the fundamental aspects of Markov chain theory on more general spaces, the chapter presents an overview of results on countable spaces, including some recent and less familiar aspects even on these spaces. Although many real systems can be described in such a framework, many other applications require continuous or even more general spaces, and there has been a flourishing of work on non-countable spaces. This chapter integrates the general state space results with topological conditions, which guarantee that the obvious (but not always true) probabilistic properties are preserved when the space has topological features.

Circulations of Markov-Chains with Continuous-Time and the Probability Interpretation of Some Determinants

The Invariance Principle for Some Class of Markov Chains

Higher Order Ergodic Markov Chains and First Passage Times

2 Circulation Distribution, Entropy Production and Irreversibility of Finite Markov Chains with Continuous Parameter

Circulation Distribution, Entropy Production And Irreversibility Of Denumerable Markov Chains

Markov Chains On Continuous State Space

Reversibility and Entropy Production of Inhomogeneous Markov Chains

On Three Classical Problems for Markov Chains with Continuous Time Parameters

Explosion, implosion, and moments of passage times for continuous-time Markov chains: a semimartingale approach

Decisive Markov Chains

Hoeffding's inequality for continuous-time Markov chains

The Derivation of Markov Chain Properties using Generalized Matrix Inverses

Markov chain properties in terms of column sums of the transition matrix

Stability Of Circulation Decompositions And Self-Organization Phenomena

Markov Chains Approximations of jump-Diffusion Quantum Trajectories

Markov chains and mappings of distributions on compact spaces II: Numerics and Conjectures

One-dimensional Continuous-Time Quantum Markov Chains: qubit probabilities and measures

Statistical Methods in Markov Chains

Stationary Probability Vectors of Higher-order Markov Chains

Markov chains and mappings of distributions on compact spaces

Markov chains: Structure and applications