Abstract:We consider the problem of estimating the asymptotic variance of a function defined on a Markov chain, an important step for statistical inference of the stationary mean. We design a novel recursive estimator that requires $O(1)$ computation at each step, does not require storing any historical samples or any prior knowledge of run-length, and has optimal $O(\frac{1}{n})$ rate of convergence for the mean-squared error (MSE) with provable finite sample guarantees. Here, $n$ refers to the total number of samples generated. Our estimator is based on linear stochastic approximation of an equivalent formulation of the asymptotic variance in terms of the solution of the Poisson equation. We generalize our estimator in several directions, including estimating the covariance matrix for vector-valued functions, estimating the stationary variance of a Markov chain, and approximately estimating the asymptotic variance in settings where the state space of the underlying Markov chain is large. We also show applications of our estimator in average reward reinforcement learning (RL), where we work with asymptotic variance as a risk measure to model safety-critical applications. We design a temporal-difference type algorithm tailored for policy evaluation in this context. We consider both the tabular and linear function approximation settings. Our work paves the way for developing actor-critic style algorithms for variance-constrained RL.

Variational Formulas of Asymptotic Variance for General Discrete-time Markov Chains

The Invariance Principle for Some Class of Markov Chains

The Variance of Partial Sums of Strong Near-Epoch Dependent Variables

Markov Chain Variance Estimation: A Stochastic Approximation Approach

Strong Approximations of Martingale Vectors and Their Applications in Markov-Chain Adaptive Designs

Some variational formulas on additive functionals of symmetric Markov chains

STRONG APPROXIMATION OF A MARKOV CHAIN

Asymptotic rates of convergence for compact reversible Markov chains

Asymptotic behaviors for Volterra type McKean-Vlasov stochastic integral equations with small noise

Theoretical guarantees for neural control variates in MCMC

Asymptotic properties for linear processes of functionals of reversible Markov Chains

Large deviations and related problems for absorbing Markov chains

A Variational Principle for Markov Processes

Reversible Markov chains: variational representations and ordering

On the asymptotic variance of reversible Markov chain without cycles

Asymptotics for A Class of Dependent Random Variables

Remarks on limit theorems for reversible Markov processes

Limit Theorems for quadratic forms of Markov Chains

A class of strong deviation theorems for the sequence of real valued random variables with respect to continuous-state non-homogeneous Markov chains