Sharp Matrix Empirical Bernstein Inequalities

Hongjian Wang,Aaditya Ramdas

2024-11-14

Abstract:We present two sharp empirical Bernstein inequalities for symmetric random matrices with bounded eigenvalues. By sharp, we mean that both inequalities adapt to the unknown variance in a tight manner: the deviation captured by the first-order $1/\sqrt{n}$ term asymptotically matches the matrix Bernstein inequality exactly, including constants, the latter requiring knowledge of the variance. Our first inequality holds for the sample mean of independent matrices, and our second inequality holds for a mean estimator under martingale dependence at stopping times.

Probability,Functional Analysis,Statistics Theory,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to provide two accurate empirical Bernstein inequalities for symmetric random matrices with bounded eigenvalues. These inequalities can adaptively capture the unknown variance, thereby estimating the common mean of independent or martingale - dependent random matrices in the non - asymptotic case.

### Specific Problem Description

1. **Background and Motivation**:
- In practical applications, although the upper bound \(B\) of a random variable can be determined almost surely, the explicit variance bound \(\sigma^{2}\) is rarely known. Therefore, the variance bound used in theoretical analysis is usually not used to actually construct the confidence interval of the mean.
- For this reason, non - asymptotic empirical Bernstein inequalities are particularly important. They only assume the almost sure upper bound \(B\) of the random variable and remain agnostic and adaptive to the true variance \(\text{Var}(X_{i})\).

2. **Existing Methods and Their Limitations**:
- Scalar empirical Bernstein inequalities are mainly based on two techniques: the union bound and the concentration inequality of the sample variance, and the self - normalizing martingale technique.
- Existing matrix Bernstein inequalities (such as Tropp's results) provide exponential concentration inequalities, but do not explicitly give empirical Bernstein inequalities.

3. **Main Contributions of the Paper**:
- Propose two new matrix empirical Bernstein inequalities, based on the union - bound method and the self - normalizing martingale technique respectively.
- These inequalities are applicable not only to independent matrices but also to martingale - dependent matrices at stopping times.
- The deviation bounds of the inequalities match those of the Oracle Bernstein inequality with known variance in the large - sample limit, and are therefore "sharp".

### Specific Results of the Paper

- **The First Inequality** (Based on the Union - Bound Method):
\[
P\left(\lambda_{\max}\left(\bar{X}_{n}-M\right)\geq D_{\text{meb}1_{n}}\right)\leq\alpha
\]
where
\[
D_{\text{meb}1_{n}}=\sqrt{\frac{2\log n d}{(n - 1)\alpha}\left(\|\hat{V}_{n}\|^{1/2}+\sqrt{\frac{2\log(2d/\alpha)}{n}\|\hat{V}_{n}\|}\wedge\left(\frac{2\log(2d/\alpha)}{n}\right)^{1/4}\right)}+\frac{\log n d}{(n - 1)\alpha}
\]

- **The Second Inequality** (Based on the Self - Normalizing Martingale Technique):
\[
P\left(\lambda_{\max}\left(\hat{M}_{n}-M\right)\geq\sqrt{\frac{2\log(d/\alpha)v_{n,\alpha}}{n}}\right)\leq\alpha
\]
where
\[
v_{n,\alpha}=\left(\frac{\log(d/\alpha)+\lambda_{\max}\left(\sum_{i = 1}^{n}\psi_{E}(\gamma_{i})(X_{i}-X_{i - 1})^{2}\right)}{\sum_{i = 1}^{n}\gamma_{i}}\right)
\]

Both of these inequalities reach the same optimal deviation bound in the large - sample limit, that is:
\[
\lim_{n\rightarrow\infty}\sqrt{n}D_{\text{meb}1_{n}}=\sqrt{2\log(d/\alpha)}