Abstract:This paper studies Kernel Density Estimation for a high-dimensional distribution $\rho(x)$. Traditional approaches have focused on the limit of large number of data points $n$ and fixed dimension $d$. We analyze instead the regime where both the number $n$ of data points $y_i$ and their dimensionality $d$ grow with a fixed ratio $\alpha=(\log n)/d$. Our study reveals three distinct statistical regimes for the kernel-based estimate of the density $\hat \rho_h^{\mathcal {D}}(x)=\frac{1}{n h^d}\sum_{i=1}^n K\left(\frac{x-y_i}{h}\right)$, depending on the bandwidth $h$: a classical regime for large bandwidth where the Central Limit Theorem (CLT) holds, which is akin to the one found in traditional approaches. Below a certain value of the bandwidth, $h_{CLT}(\alpha)$, we find that the CLT breaks down. The statistics of $\hat\rho_h^{\mathcal {D}}(x)$ for a fixed $x$ drawn from $\rho(x)$ is given by a heavy-tailed distribution (an alpha-stable distribution). In particular below a value $h_G(\alpha)$, we find that $\hat\rho_h^{\mathcal {D}}(x)$ is governed by extreme value statistics: only a few points in the database matter and give the dominant contribution to the density estimator. We provide a detailed analysis for high-dimensional multivariate Gaussian data. We show that the optimal bandwidth threshold based on Kullback-Leibler divergence lies in the new statistical regime identified in this paper. As known by practitioners, when decreasing the bandwidth a Kernel-estimated estimated changes from a smooth curve to a collections of peaks centred on the data points. Our findings reveal that this general phenomenon is related to sharp transitions between phases characterized by different statistical properties, and offer new insights for Kernel density estimation in high-dimensional settings.

Reducing bias in nonparametric density estimation via bandwidth dependent kernels: $L_1$ view

Limit theorems for kernel density estimators under dependent samples

Reduced bias nonparametric lifetime density and hazard estimation

Efficient Bandwidth Selection for Kernel Density Estimation

Design of Two MUSIC-like Estimators Based on Bias Minimization

A weighted least-squares cross-validation bandwidth selector for kernel density estimation

Bandwidth Selection for Kernel Density Estimation with a Markov Chain Monte Carlo Sample

Kernel Density Estimators in Large Dimensions

Uniform Convergence Rate of the Kernel Density Estimator Adaptive to Intrinsic Volume Dimension

Variable bandwidth kernel regression estimation

Unified treatment of the asymptotics of asymmetric kernel density estimators

Pointwise adaptive kernel density estimation under local approximate differential privacy

Multivariate Locally Adaptive Kernel Density Estimation

Beta Kernel Estimators for Density Functions

A note on the convergence rate of the kernel density estimator of the mode

Bayesian Inference for $k$-Monotone Densities with Applications to Multiple Testing

Beyond Smoothness: Incorporating Low-Rank Analysis into Nonparametric Density Estimation

On the Decreasing Power of Kernel and Distance Based Nonparametric Hypothesis Tests in High Dimensions

Tuning Selection for Two-Scale Kernel Density Estimators.

Variational Weighting for Kernel Density Ratios

Numerical performance of Penalized Comparison to Overfitting for multivariate kernel density estimation