Abstract:Statistical divergences are ubiquitous in machine learning as tools for measuring discrepancy between probability distributions. As these applications inherently rely on approximating distributions from samples, we consider empirical approximation under two popular $f$-divergences: the total variation (TV) distance and the $\chi^2$-divergence. To circumvent the sensitivity of these divergences to support mismatch, the framework of Gaussian smoothing is adopted. We study the limit distributions of $\sqrt{n}\delta_{\mathsf{TV}}(P_n\ast\mathcal{N},P\ast\mathcal{N})$ and $n\chi^2(P_n\ast\mathcal{N}\|P\ast\mathcal{N})$, where $P_n$ is the empirical measure based on $n$ independently and identically distributed (i.i.d.) observations from $P$, $\mathcal{N}_\sigma:=\mathcal{N}(0,\sigma^2\mathrm{I}_d)$, and $\ast$ stands for convolution. In arbitrary dimension, the limit distributions are characterized in terms of Gaussian process on $\mathbb{R}^d$ with covariance operator that depends on $P$ and the isotropic Gaussian density of parameter $\sigma$. This, in turn, implies optimality of the $n^{-1/2}$ expected value convergence rates recently derived for $\delta_{\mathsf{TV}}(P_n\ast\mathcal{N},P\ast\mathcal{N})$ and $\chi^2(P_n\ast\mathcal{N}\|P\ast\mathcal{N})$. These strong statistical guarantees promote empirical approximation under Gaussian smoothing as a potent framework for learning and inference based on high-dimensional data.

On the tensorization of the variational distance

Total Variation Distance Meets Probabilistic Inference

Total Variation Distance for Product Distributions is $\#\mathsf{P}$-Complete

Improved bounds for the total variation distance between stochastic polynomials

Limit Distribution for Smooth Total Variation and $χ^2$-Divergence in High Dimensions

Total Variation Meets Differential Privacy

From Soft-Minoration to Information-Constrained Optimal Transport and Spiked Tensor Models

Stable central limit theorem in total variation distance

Lower and Upper Bounds on the VC-Dimension of Tensor Network Models

A short note on an inequality between KL and TV

Probabilistic bounds on best rank-1 approximation ratio

Wasserstein and total variation distance between marginals of Lévy processes

Entropy bounds for the absolute convex hull of tensors

A simple polynomial-time approximation algorithm for the total variation distance between two product distributions

Distance multivariance: New dependence measures for random vectors

On a variance dependent Dvoretzky-Kiefer-Wolfowitz inequality

Variational Estimates for Bilinear Ergodic Averages Along Sublinear Sequences

Overiteration of $d$-variate tensor product Bernstein operators: a quantitative result

Stochastic optimal transport in Banach Spaces for regularized estimation of multivariate quantiles

Guarantees of Total Variation Minimization for Signal Recovery

Quantitative stability for stochastic tensor variational inequalities