Abstract:We show that the minimax sample complexity for estimating the pseudo-spectral gap $\gamma_{\mathsf{ps}}$ of an ergodic Markov chain in constant multiplicative error is of the order of $$\tilde{\Theta}\left( \frac{1}{\gamma_{\mathsf{ps}} \pi_{\star}} \right),$$ where $\pi_\star$ is the minimum stationary probability, recovering the known bound in the reversible setting for estimating the absolute spectral gap [Hsu et al., 2019], and resolving an open problem of Wolfer and Kontorovich [2019]. Furthermore, we strengthen the known empirical procedure by making it fully-adaptive to the data, thinning the confidence intervals and reducing the computational complexity. Along the way, we derive new properties of the pseudo-spectral gap and introduce the notion of a reversible dilation of a stochastic matrix.
What problem does this paper attempt to address?
This paper aims to solve the problem of estimating the pseudo - spectral gap (\(\gamma_{\text{ps}}\)) in non - reversible Markov chains. Specifically, the goals of the paper are:
1. **Estimate the sample complexity of the pseudo - spectral gap**: The author shows that the minimax sample complexity for estimating the pseudo - spectral gap \(\gamma_{\text{ps}}\) of a non - reversible Markov chain under a constant multiplicative error is \(\tilde{\Theta}\left(\frac{1}{\gamma_{\text{ps}}\pi^*}\right)\), where \(\pi^*\) is the smallest stationary probability. This recovers the known bound for estimating the absolute spectral gap \(\gamma^*\) in the reversible case [Hsu et al., 2019] and solves an open problem proposed by Wolfer and Kontorovich [2019].
2. **Improve the empirical estimation method**: The paper strengthens the known empirical procedures by making the estimation process fully adapt to the data, narrowing the confidence interval, and reducing the computational complexity.
3. **Introduce new concepts and properties**: The author derives new properties of the pseudo - spectral gap and introduces the concept of reversible dilation of a random matrix.
### Main contributions
- **Theorem 2.1**: For any additive error \(\epsilon\), the upper bound of the sample complexity for estimating \(\gamma_{\text{ps}}\) is \(\tilde{O}\left(\frac{1}{\epsilon^2\pi^*\gamma_{\text{ps}}}\right)\).
- **Theorem 2.2**: For a constant multiplicative error, the upper bound of the sample complexity for estimating \(\gamma_{\text{ps}}\) is \(\tilde{O}\left(\frac{1}{\pi^*\gamma_{\text{ps}}}\right)\). This result is stronger than that in Wolfer and Kontorovich [2019] and is consistent with the known results in the reversible case [Hsu et al., 2019].
- **Theorem 2.3**: For any small multiplicative error \(\epsilon\), the upper bound of the sample complexity for estimating \(\gamma_{\text{ps}}\) is \(\tilde{O}\left(\frac{1}{\epsilon^2\pi^*\gamma_{\text{ps}}^3}\right)\).
- **Definition 3.1**: Introduces the reversible dilation of a Markov chain, whose spectral properties are closely related to Fill's multiple reversibilization and has a lower computational cost when the stationary distribution is known.
- **Theorem 3.1**: Under non - trivial conditions, improves the width of the confidence interval in Wolfer and Kontorovich [2019] and makes the whole process fully adapt to the data.
### Background and related work
- **Reversible case**: Hsu et al. [2015] began to study estimating the absolute spectral gap of a reversible Markov chain from a single trajectory and gave the upper and lower bounds under multiplicative error. Subsequent studies further optimized these results [Levin and Peres, 2016; Hsu et al., 2019; Wolfer and Kontorovich, 2019].
- **Non - reversible case**: Wolfer and Kontorovich [2019] first studied the problem of estimating the pseudo - spectral gap of a non - reversible Markov chain, gave the upper bound of the sample complexity, and constructed an empirical confidence interval. This paper further improves these results on this basis.
### Applications
- **MCMC diagnosis**: Non - reversible Markov chains are receiving increasing attention in acceleration methods because they may have better mixing properties and smaller asymptotic variances.
- **Reinforcement learning**: In reinforcement learning, it is usually assumed that the mixing parameters of the Markov decision process are bounded.