Abstract:This paper studies Kernel Density Estimation for a high-dimensional distribution $\rho(x)$. Traditional approaches have focused on the limit of large number of data points $n$ and fixed dimension $d$. We analyze instead the regime where both the number $n$ of data points $y_i$ and their dimensionality $d$ grow with a fixed ratio $\alpha=(\log n)/d$. Our study reveals three distinct statistical regimes for the kernel-based estimate of the density $\hat \rho_h^{\mathcal {D}}(x)=\frac{1}{n h^d}\sum_{i=1}^n K\left(\frac{x-y_i}{h}\right)$, depending on the bandwidth $h$: a classical regime for large bandwidth where the Central Limit Theorem (CLT) holds, which is akin to the one found in traditional approaches. Below a certain value of the bandwidth, $h_{CLT}(\alpha)$, we find that the CLT breaks down. The statistics of $\hat\rho_h^{\mathcal {D}}(x)$ for a fixed $x$ drawn from $\rho(x)$ is given by a heavy-tailed distribution (an alpha-stable distribution). In particular below a value $h_G(\alpha)$, we find that $\hat\rho_h^{\mathcal {D}}(x)$ is governed by extreme value statistics: only a few points in the database matter and give the dominant contribution to the density estimator. We provide a detailed analysis for high-dimensional multivariate Gaussian data. We show that the optimal bandwidth threshold based on Kullback-Leibler divergence lies in the new statistical regime identified in this paper. As known by practitioners, when decreasing the bandwidth a Kernel-estimated estimated changes from a smooth curve to a collections of peaks centred on the data points. Our findings reveal that this general phenomenon is related to sharp transitions between phases characterized by different statistical properties, and offer new insights for Kernel density estimation in high-dimensional settings.

Bandwidth Selection for Kernel Density Estimation with a Markov Chain Monte Carlo Sample

Efficient Bandwidth Selection for Kernel Density Estimation

A weighted least-squares cross-validation bandwidth selector for kernel density estimation

Limit theorems for kernel density estimators under dependent samples

Consistency of Kernel Density Estimators for Causal Processes

Multivariate Statistical Process Monitoring Using Kernel Density Estimation

Kernel Averaging Estimators

Multivariate Locally Adaptive Kernel Density Estimation

Kernel Density Estimators in Large Dimensions

Adaptive Manifold Density Estimation

Band Width Selection for High Dimensional Covariance Matrix Estimation

Unified treatment of the asymptotics of asymmetric kernel density estimators

Variable bandwidth kernel regression estimation

Reducing bias in nonparametric density estimation via bandwidth dependent kernels: $L_1$ view

Sequential Cross-Validated Bandwidth Selection Under Dependence and Anscombe-Type Extensions to Random Time Horizons

A Vine-Copula Based Adaptive Mcmc Sampler for Efficient Inference of Dynamical Systems

Numerical performance of Penalized Comparison to Overfitting for multivariate kernel density estimation

Plug-in bandwidth selection rules for the kernel quantile estimator

Bayesian bandwidth estimation and semi-metric selection for a functional partial linear model with unknown error density

Density Estimation by Monte Carlo and Quasi-Monte Carlo

Novel kernel density estimator based on ensemble unbiased cross-validation