Compressing redundant information in Markov chains

Giacomo Aletti

DOI: https://doi.org/10.48550/arXiv.math/0605099

2006-05-03

Probability

Abstract:Given a strongly stationary Markov chain and a finite set of stopping rules, we prove the existence of a polynomial algorithm which projects the Markov chain onto a minimal Markov chain without redundant information. Markov complexity is hence defined and tested on some classical problems.

What problem does this paper attempt to address?

Information-Preserving Markov Aggregation

Bernhard C. Geiger,Christoph Temmel

DOI: https://doi.org/10.48550/arXiv.1304.0920

2013-07-24

Abstract:We present a sufficient condition for a non-injective function of a Markov chain to be a second-order Markov chain with the same entropy rate as the original chain. This permits an information-preserving state space reduction by merging states or, equivalently, lossless compression of a Markov source on a sample-by-sample basis. The cardinality of the reduced state space is bounded from below by the node degrees of the transition graph associated with the original Markov chain. We also present an algorithm listing all possible information-preserving state space reductions, for a given transition graph. We illustrate our results by applying the algorithm to a bi-gram letter model of an English text.

Information Theory
The Markov-Chain Polytope with Applications

Mordecai J. Golin,Albert John Lalim Patupat

2024-08-17

Abstract:This paper addresses the problem of finding a minimum-cost $m$-state Markov chain $(S_0,\ldots,S_{m-1})$ in a large set of chains. The chains studied have a reward associated with each state. The cost of a chain is its "gain", i.e., its average reward under its stationary distribution. Specifically, for each $k=0,\ldots,m-1$ there is a known set ${\mathbb S}_k$ of type-$k$ states. A permissible Markov chain contains exactly one state of each type; the problem is to find a minimum-cost permissible chain. The original motivation was to find a cheapest binary AIFV-$m$ lossless code on a source alphabet of size $n$. Such a code is an $m$-tuple of trees, in which each tree can be viewed as a Markov Chain state. This formulation was then used to address other problems in lossless compression. The known solution techniques for finding minimum-cost Markov chains were iterative and ran in exponential time. This paper shows how to map every possible type-$k$ state into a type-$k$ hyperplane and then define a "Markov Chain Polytope" as the lower envelope of all such hyperplanes. Finding a minimum-cost Markov chain can then be shown to be equivalent to finding a "highest" point on this polytope. The local optimization procedures used in the previous iterative algorithms are shown to be separation oracles for this polytope. Since these were often polynomial time, an application of the Ellipsoid method immediately leads to polynomial time algorithms for these problems.

Information Theory
Compressibility and Stochastic Stability of Monotone Markov Chain

Sergey Foss,Michael Scheutzow

2024-03-22

Abstract:For a stochastically monotone Markov chain taking values in a Polish space, we present a number of conditions for existence and for uniqueness of its stationary regime, as well as for closeness of its transient trajectories. In particular, we generalise a basic result by Bhattacharya and Majumdar (2007) where a certain form of mixing, or swap condition was assumed uniformly over the state space. We do not rely on continuity properties of transition probabilities.

Probability
Beyond Decisiveness of Infinite Markov Chains

Benoît Barbot,Patricia Bouyer,Serge Haddad

2024-09-27

Abstract:Verification of infinite-state Markov chains is still a challenge despite several fruitful numerical or statistical approaches. For decisive Markov chains, there is a simple numerical algorithm that frames the reachability probability as accurately as required (however with an unknown complexity). On the other hand when applicable, statistical model checking is in most of the cases very efficient. Here we study the relation between these two approaches showing first that decisiveness is a necessary and sufficient condition for almost sure termination of statistical model checking. Afterwards we develop an approach with application to both methods that substitutes to a non decisive Markov chain a decisive Markov chain with the same reachability probability. This approach combines two key ingredients: abstraction and importance sampling (a technique that was formerly used for efficiency). We develop this approach on a generic formalism called layered Markov chain (LMC). Afterwards we perform an empirical study on probabilistic pushdown automata (an instance of LMC) to understand the complexity factors of the statistical and numerical algorithms. To the best of our knowledge, this prototype is the first implementation of the deterministic algorithm for decisive Markov chains and required us to solve several qualitative and numerical issues.

Logic in Computer Science,Formal Languages and Automata Theory
Minimax redundancy for Markov chains with large state space

Kedar Shriram Tatwawadi,Jiantao Jiao,Tsachy Weissman

DOI: https://doi.org/10.48550/arXiv.1805.01355

2018-05-06

Abstract:For any Markov source, there exist universal codes whose normalized codelength approaches the Shannon limit asymptotically as the number of samples goes to infinity. This paper investigates how fast the gap between the normalized codelength of the "best" universal compressor and the Shannon limit (i.e. the compression redundancy) vanishes non-asymptotically in terms of the alphabet size and mixing time of the Markov source. We show that, for Markov sources whose relaxation time is at least $1 + \frac{(2+c)}{\sqrt{k}}$, where $k$ is the state space size (and $c>0$ is a constant), the phase transition for the number of samples required to achieve vanishing compression redundancy is precisely $\Theta(k^2)$.

Information Theory,Signal Processing
Polynomials Associated with Finite Markov Chains

Philippe Biane

DOI: https://doi.org/10.1007/978-3-319-18585-9_12

2015-01-01

Lecture notes in mathematics

Abstract:Given a finite Markov chain, we investigate the first minors of the transition matrix of a lifting of this Markov chain to covering trees. In a simple case we exhibit a nice factorisation of these minors, and we conjecture that it holds more generally.

mathematics
Constructing Markov chains with given dependence and marginal stationary distributions

Tomonari Sei

2024-07-25

Abstract:A method of constructing Markov chains on finite state spaces is provided. The chain is specified by three constraints: stationarity, dependence and marginal distributions. The generalized Pythagorean theorem in information geometry plays a central role in the construction. An algorithm for obtaining the desired Markov chain is described. Integer-valued autoregressive processes are considered for illustration.

Statistics Theory
Analytic-geometric methods for finite Markov chains with applications to quasi-stationarity

Persi Diaconis,Kelsey Houston-Edwards,Laurent Saloff-Coste

DOI: https://doi.org/10.48550/arXiv.1906.04877

2019-06-12

Abstract:For a relatively large class of well-behaved absorbing (or killed) finite Markov chains, we give detailed quantitative estimates regarding the behavior of the chain before it is absorbed (or killed). Typical examples are random walks on box-like finite subsets of the square lattice $\mathbb Z^d$ absorbed (or killed) at the boundary. The analysis is based on Poincaré, Nash, and Harnack inequalities, moderate growth, and on the notions of John and inner-uniform domains.

Probability
Some Limit Properties of Markov Chains Induced by Stochastic Recursive Algorithms

Abhishek Gupta,Hao Chen,Jianzong Pi,Gaurav Tendolkar

DOI: https://doi.org/10.48550/arXiv.1904.10778

2020-07-24

Abstract:Recursive stochastic algorithms have gained significant attention in the recent past due to data driven applications. Examples include stochastic gradient descent for solving large-scale optimization problems and empirical dynamic programming algorithms for solving Markov decision problems. These recursive stochastic algorithms approximate certain contraction operators and can be viewed within the framework of iterated random operators. Accordingly, we consider iterated random operators over a Polish space that simulate iterated contraction operator over that Polish space. Assume that the iterated random operators are indexed by certain batch sizes such that as batch sizes grow to infinity, each realization of the random operator converges (in some sense) to the contraction operator it is simulating. We show that starting from the same initial condition, the distribution of the random sequence generated by the iterated random operators converges weakly to the trajectory generated by the contraction operator. We further show that under certain conditions, the time average of the random sequence converges to the spatial mean of the invariant distribution. We then apply these results to logistic regression, empirical value iteration, and empirical Q value iteration for finite state finite action MDPs to illustrate the general theory develop here.

Machine Learning,Systems and Control,Optimization and Control,Probability
Geometry and Duality of Alternating Markov Chains

Deven Mithal,Lorenzo Orecchia

2024-10-17

Abstract:We realize the half-steps of a general class of Markov chains as alternating projections with respect to the reverse Kullback-Leibler divergence between sets of joint probability distributions which admit an information geometrical property known as autoparallelism. We exhibit both a weak and strong duality between the Markov chains defined by the even and odd half-steps of the alternating projection scheme, which manifests as an equivalence of entropy decay statement regarding the chains, with a precise characterization of the entropy decrease at each half-step. We apply this duality to several Markov chains of interest, and obtain either new results or short, alternative proofs for mixing bounds on the dual chain, with the Swendsen-Wang dynamics serving as a key example. Additionally, we draw parallels between the half-steps of the relevant Markov chains and the Sinkhorn algorithm from the field of entropically regularized optimal transport, which are unified by the perspective of alternating projections with respect to an \alpha-divergence on the probability simplex.

Probability,Information Theory
Reversible Mcmc on Markov Equivalence Classes of Sparse Directed Acyclic Graphs

Yangbo He,Jinzhu Jia,Bin Yu

DOI: https://doi.org/10.1214/13-aos1125

2013-01-01

Abstract:Graphical models are popular statistical tools which are used to represent dependent or causal complex systems. Statistically equivalent causal or directed graphical models are said to belong to a Markov equivalent class. It is of great interest to describe and understand the space of such classes. However, with currently known algorithms, sampling over such classes is only feasible for graphs with fewer than approximately 20 vertices. In this paper, we design reversible irreducible Markov chains on the space of Markov equivalent classes by proposing a perfect set of operators that determine the transitions of the Markov chain. The stationary distribution of a proposed Markov chain has a closed form and can be computed easily. Specifically, we construct a concrete perfect set of operators on sparse Markov equivalence classes by introducing appropriate conditions on each possible operator. Algorithms and their accelerated versions are provided to efficiently generate Markov chains and to explore properties of Markov equivalence classes of sparse directed acyclic graphs (DAGs) with thousands of vertices. We find experimentally that in most Markov equivalence classes of sparse DAGs, (1) most edges are directed, (2) most undirected subgraphs are small and (3) the number of these undirected subgraphs grows approximately linearly with the number of vertices.
The importance Markov chain

Charly Andral,Randal Douc,Hugo Marival,Christian P. Robert

DOI: https://doi.org/10.1016/j.spa.2024.104316

IF: 1.43

2024-02-01

Stochastic Processes and their Applications

Abstract:The Importance Markov chain is a novel algorithm bridging the gap between rejection sampling and importance sampling, moving from one to the other through a tuning parameter. Based on a modified sample of an instrumental Markov chain targeting an instrumental distribution (typically via a MCMC kernel), the Importance Markov chain produces an extended Markov chain where the marginal distribution of the first component converges to the target distribution. For example, when targeting a multimodal distribution, the instrumental distribution can be chosen as a tempered version of the target which allows the algorithm to explore its modes more efficiently. We obtain a Law of Large Numbers and a Central Limit Theorem as well as geometric ergodicity for this extended kernel under mild assumptions on the instrumental kernel. Computationally, the algorithm is easy to implement and preexisting librairies can be used to sample from the instrumental distribution.

statistics & probability
Markov chains and mappings of distributions on compact spaces

David J. Aldous,Shi Feng

2024-04-01

Abstract:Consider a compact metric space $S$ and a pair $(j,k)$ with $k \ge 2$ and $1 \le j \le k$. For any probability distribution $\theta \in P(S)$, define a Markov chain on $S$ by: from state $s$, take $k$ i.i.d. ($\theta$) samples, and jump to the $j$'th closest. Such a chain converges in distribution to a unique stationary distribution, say $\pi_{j,k}(\theta)$. So this defines a mapping $\pi_{j,k}: P(S) \to P(S)$. What happens when we iterate this mapping? In particular, what are the fixed points of this mapping? We present a few rigorous results, to complement our extensive simulation study elsewhere.

Probability,Metric Geometry
Algorithms for Markov Binomial Chains

Alejandro Alarcón Gonzalez,Niel Hens,Tim Leys,Guillermo A. Pérez

2024-08-09

Abstract:We study algorithms to analyze a particular class of Markov population processes that is often used in epidemiology. More specifically, Markov binomial chains are the model that arises from stochastic time-discretizations of classical compartmental models. In this work we formalize this class of Markov population processes and focus on the problem of computing the expected time to termination in a given such model. Our theoretical contributions include proving that Markov binomial chains whose flow of individuals through compartments is acyclic almost surely terminate. We give a PSPACE algorithm for the problem of approximating the time to termination and a direct algorithm for the exact problem in the Blum-Shub-Smale model of computation. Finally, we provide a natural encoding of Markov binomial chains into a common input language for probabilistic model checkers. We implemented the latter encoding and present some initial empirical results showcasing what formal methods can do for practicing epidemilogists.

Logic in Computer Science
Markov chains: Structure and applications

R.L. Tweedie

DOI: https://doi.org/10.1016/s0169-7161(01)19025-5

2001-01-01

Handbook of Statistics

Abstract:Publisher The classical theory of Markov chains originally dealt only with chains on finite or countable spaces. In order to establish the fundamental aspects of Markov chain theory on more general spaces, the chapter presents an overview of results on countable spaces, including some recent and less familiar aspects even on these spaces. Although many real systems can be described in such a framework, many other applications require continuous or even more general spaces, and there has been a flourishing of work on non-countable spaces. This chapter integrates the general state space results with topological conditions, which guarantee that the obvious (but not always true) probabilistic properties are preserved when the space has topological features.
On the Complexity of Computing Maximum Entropy for Markovian Models.

Taolue Chen,Tingting Han

DOI: https://doi.org/10.4230/lipics.fsttcs.2014.571

2014-01-01

Abstract:We investigate the complexity of computing entropy of various Markovian models including Markov Chains (MCs), Interval Markov Chains (IMCs) and Markov Decision Processes (MDPs). We consider both entropy and entropy rate for general MCs, and study two algorithmic questions, i.e., entropy approximation problem and entropy threshold problem. The former asks for an approximation of the entropy/entropy rate within a given precision, whereas the latter aims to decide whether they exceed a given threshold. We give polynomial-time algorithms for the approximation problem, and show the threshold problem is in P CH3 (hence in PSPACE) and in P assuming some number-theoretic conjectures. Furthermore, we study both questions for IMCs and MDPs where we aim to maximise the entropy/entropy rate among an infinite family of MCs associated with the given model. We give various conditional decidability results for the threshold problem, and show the approximation problem is solvable in polynomial-time via convex programming.
Quasi-stationary distributions in reducible state spaces

Nicolas Champagnat,Denis Villemonais

2024-07-09

Abstract:We study quasi-stationary distributions and quasi-limiting behavior of Markov chains in general reducible state spaces with absorption. We propose a set of assumptions dealing with particular situations where the state space can be decomposed into three subsets between which communication is only possible in a single direction. These assumptions allow us to characterize the exponential order of magnitude and the exact polynomial correction, called polynomial convergence parameter, for the leading order term of the semigroup for large time. They also provide explicit convergence speeds to this leading order term. We apply these results to general Markov chains with finitely or denumerably many communication classes using a specific induction over the communication classes of the chain. We are able to explicitely characterize the polynomial convergence parameter, to determine the complete set of quasistationary distributions and to provide explicit estimates for the speed of convergence to quasi-limiting distributions in the case of finitely many communication classes. We conclude with an application of these results to the case of denumerable state spaces, where we are able to prove that, in general, there is existence of a quasi-stationary distribution without assuming irreducibility before absorption. This actually holds true assuming only aperiodicity, the existence of a Lyapunov function and the existence of a point in the state space from which the return time is finite with positive probability.

Probability
Graph-combinatorial approach for large deviations of Markov chains

Giorgio Carugno,Pierpaolo Vivo,Francesco Coghi

DOI: https://doi.org/10.1088/1751-8121/ac79e6

2022-06-03

Abstract:We consider discrete-time Markov chains and study large deviations of the pair empirical occupation measure, which is useful to compute fluctuations of pure-additive and jump-type observables. We provide an exact expression for the finite-time moment generating function, which is split in cycles and paths contributions, and scaled cumulant generating function of the pair empirical occupation measure via a graph-combinatorial approach. The expression obtained allows us to give a physical interpretation of interaction and entropic terms, and of the Lagrange multipliers, and may serve as a starting point for sub-leading asymptotics. We illustrate the use of the method for a simple two-state Markov chain.

Statistical Mechanics,Probability
Characterization of cutoff for reversible Markov chains

Riddhipratim Basu,Jonathan Hermon,Yuval Peres

DOI: https://doi.org/10.1214/16-AOP1090

2015-02-18

Abstract:A sequence of Markov chains is said to exhibit (total variation) cutoff if the convergence to stationarity in total variation distance is abrupt. We consider reversible lazy chains. We prove a necessary and sufficient condition for the occurrence of the cutoff phenomena in terms of concentration of hitting time of "worst" (in some sense) sets of stationary measure at least $\alpha$, for some $\alpha \in (0,1)$. We also give general bounds on the total variation distance of a reversible chain at time $t$ in terms of the probability that some "worst" set of stationary measure at least $\alpha$ was not hit by time $t$. As an application of our techniques we show that a sequence of lazy Markov chains on finite trees exhibits a cutoff iff the ratio of their relaxation-times and their (lazy) mixing-times tends to 0.

Probability
Information Theoretic Properties of Markov Random Fields, and their Algorithmic Applications

Linus Hamilton,Frederic Koehler,Ankur Moitra

DOI: https://doi.org/10.48550/arXiv.1705.11107

2017-05-31

Abstract:Markov random fields area popular model for high-dimensional probability distributions. Over the years, many mathematical, statistical and algorithmic problems on them have been studied. Until recently, the only known algorithms for provably learning them relied on exhaustive search, correlation decay or various incoherence assumptions. Bresler gave an algorithm for learning general Ising models on bounded degree graphs. His approach was based on a structural result about mutual information in Ising models. Here we take a more conceptual approach to proving lower bounds on the mutual information through setting up an appropriate zero-sum game. Our proof generalizes well beyond Ising models, to arbitrary Markov random fields with higher order interactions. As an application, we obtain algorithms for learning Markov random fields on bounded degree graphs on $n$ nodes with $r$-order interactions in $n^r$ time and $\log n$ sample complexity. The sample complexity is information theoretically optimal up to the dependence on the maximum degree. The running time is nearly optimal under standard conjectures about the hardness of learning parity with noise.

Machine Learning,Data Structures and Algorithms,Information Theory,Statistics Theory

Compressing redundant information in Markov chains

Information-Preserving Markov Aggregation

The Markov-Chain Polytope with Applications

Compressibility and Stochastic Stability of Monotone Markov Chain

Beyond Decisiveness of Infinite Markov Chains

Minimax redundancy for Markov chains with large state space

Polynomials Associated with Finite Markov Chains

Constructing Markov chains with given dependence and marginal stationary distributions

Analytic-geometric methods for finite Markov chains with applications to quasi-stationarity

Some Limit Properties of Markov Chains Induced by Stochastic Recursive Algorithms

Geometry and Duality of Alternating Markov Chains

Reversible Mcmc on Markov Equivalence Classes of Sparse Directed Acyclic Graphs

The importance Markov chain

Markov chains and mappings of distributions on compact spaces

Algorithms for Markov Binomial Chains

Markov chains: Structure and applications

On the Complexity of Computing Maximum Entropy for Markovian Models.

Quasi-stationary distributions in reducible state spaces

Graph-combinatorial approach for large deviations of Markov chains

Characterization of cutoff for reversible Markov chains

Information Theoretic Properties of Markov Random Fields, and their Algorithmic Applications