Abstract:We present differentially private algorithms for high-dimensional mean estimation. Previous private estimators on distributions over $\mathbb{R}^d$ suffer from a curse of dimensionality, as they require $\Omega(d^{1/2})$ samples to achieve non-trivial error, even in cases where $O(1)$ samples suffice without privacy. This rate is unavoidable when the distribution is isotropic, namely, when the covariance is a multiple of the identity matrix, or when accuracy is measured with respect to the affine-invariant Mahalanobis distance. Yet, real-world data is often highly anisotropic, with signals concentrated on a small number of principal components. We develop estimators that are appropriate for such signals$\unicode{x2013}$our estimators are $(\varepsilon,\delta)$-differentially private and have sample complexity that is dimension-independent for anisotropic subgaussian distributions. Given $n$ samples from a distribution with known covariance-proxy $\Sigma$ and unknown mean $\mu$, we present an estimator $\hat{\mu}$ that achieves error $\|\hat{\mu}-\mu\|_2\leq \alpha$, as long as $n\gtrsim\mathrm{tr}(\Sigma)/\alpha^2+ \mathrm{tr}(\Sigma^{1/2})/(\alpha\varepsilon)$. In particular, when $\pmb{\sigma}^2=(\sigma_1^2, \ldots, \sigma_d^2)$ are the singular values of $\Sigma$, we have $\mathrm{tr}(\Sigma)=\|\pmb{\sigma}\|_2^2$ and $\mathrm{tr}(\Sigma^{1/2})=\|\pmb{\sigma}\|_1$, and hence our bound avoids dimension-dependence when the signal is concentrated in a few principal components. We show that this is the optimal sample complexity for this task up to logarithmic factors. Moreover, for the case of unknown covariance, we present an algorithm whose sample complexity has improved dependence on the dimension, from $d^{1/2}$ to $d^{1/4}$.

Sharper dimension-free bounds on the Frobenius distance between sample covariance and its expectation

Dimension-free Structured Covariance Estimation

Convergence Rates of Spectral Distributions of Large Sample Covariance Matrices

A Note on the Convergence Rate of the Spectral Distributions of Large Sample Covariance Matrices

Estimation of Shortest Path Covariance Matrices

Beyond Catoni: Sharper Rates for Heavy-Tailed and Robust Mean Estimation

Better and Simpler Lower Bounds for Differentially Private Statistical Estimation

Liberating dimension and spectral norm: A universal approach to spectral properties of sample covariance matrices

Almost sure convergence of the largest and smallest eigenvalues of high-dimensional sample correlation matrices

On asymptotics of eigenvectors of large sample covariance matrix

Robust Sparse Mean Estimation via Sum of Squares

Estimation of trace functionals and spectral measures of covariance operators in Gaussian models

Almost sharp covariance and Wishart-type matrix estimation

Asymptotics of Distances Between Sample Covariance Matrices

Deterministic Equivalent of the Log-Euclidean Distance between Sample Covariance Matrices

Lower Bounds for Private Estimation of Gaussian Covariance Matrices under All Reasonable Parameter Regimes

Dimension-free Private Mean Estimation for Anisotropic Distributions

Anticoncentration and Berry--Esseen bounds for random tensors

Covariance Estimation: Optimal Dimension-free Guarantees for Adversarial Corruption and Heavy Tails

Shrinkage MMSE estimators of covariances beyond the zero-mean and stationary variance assumptions