Likelihood Ratio Gradient Estimation for Steady-State Parameters

Peter W. Glynn,Mariana Olvera-Cravioto
DOI: https://doi.org/10.48550/arXiv.1707.02659
2018-03-09
Abstract:We consider a discrete-time Markov chain $\boldsymbol{\Phi}$ on a general state-space ${\sf X}$, whose transition probabilities are parameterized by a real-valued vector $\boldsymbol{\theta}$. Under the assumption that $\boldsymbol{\Phi}$ is geometrically ergodic with corresponding stationary distribution $\pi(\boldsymbol{\theta})$, we are interested in estimating the gradient $\nabla \alpha(\boldsymbol{\theta})$ of the steady-state expectation $$\alpha(\boldsymbol{\theta}) = \pi( \boldsymbol{\theta}) f.$$ To this end, we first give sufficient conditions for the differentiability of $\alpha(\boldsymbol{\theta})$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.
Probability
What problem does this paper attempt to address?