Abstract:This paper studies Kernel Density Estimation for a high-dimensional distribution $\rho(x)$. Traditional approaches have focused on the limit of large number of data points $n$ and fixed dimension $d$. We analyze instead the regime where both the number $n$ of data points $y_i$ and their dimensionality $d$ grow with a fixed ratio $\alpha=(\log n)/d$. Our study reveals three distinct statistical regimes for the kernel-based estimate of the density $\hat \rho_h^{\mathcal {D}}(x)=\frac{1}{n h^d}\sum_{i=1}^n K\left(\frac{x-y_i}{h}\right)$, depending on the bandwidth $h$: a classical regime for large bandwidth where the Central Limit Theorem (CLT) holds, which is akin to the one found in traditional approaches. Below a certain value of the bandwidth, $h_{CLT}(\alpha)$, we find that the CLT breaks down. The statistics of $\hat\rho_h^{\mathcal {D}}(x)$ for a fixed $x$ drawn from $\rho(x)$ is given by a heavy-tailed distribution (an alpha-stable distribution). In particular below a value $h_G(\alpha)$, we find that $\hat\rho_h^{\mathcal {D}}(x)$ is governed by extreme value statistics: only a few points in the database matter and give the dominant contribution to the density estimator. We provide a detailed analysis for high-dimensional multivariate Gaussian data. We show that the optimal bandwidth threshold based on Kullback-Leibler divergence lies in the new statistical regime identified in this paper. As known by practitioners, when decreasing the bandwidth a Kernel-estimated estimated changes from a smooth curve to a collections of peaks centred on the data points. Our findings reveal that this general phenomenon is related to sharp transitions between phases characterized by different statistical properties, and offer new insights for Kernel density estimation in high-dimensional settings.

The phase diagram of kernel interpolation in large dimensions

Kernel interpolation generalizes poorly

Kernel Interpolation of High Dimensional Scattered Data

Distance-Based Classifier Via the Kernel Trick.

On the convergence of generalized kernel-based interpolation by greedy data selection algorithms

Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensions

Optimal Rate of Kernel Regression in Large Dimensions

A hybrid stochastic interpolation and compression method for kernel matrices

On the Optimality of Target-Data-Dependent Kernel Greedy Interpolation in Sobolev Reproducing Kernel Hilbert Spaces

Scattered Data Interpolation on Embedded Submanifolds with Restricted Positive Definite Kernels: Sobolev Error Estimates

Polyharmonic and Related Kernels on Manifolds: Interpolation and Approximation

The Convergence of the Geometric Interpolation Algorithm.

High-Dimensional Gaussian Process Regression with Soft Kernel Interpolation

Product kernels are efficient and flexible tools for high-dimensional scattered data interpolation

Kernel Density Estimators in Large Dimensions

Universality of kernel random matrices and kernel regression in the quadratic regime

Harmless interpolation in regression and classification with structured features

On the Pinsker bound of inner product kernel regression in large dimensions

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalization