Abstract:This paper studies Kernel Density Estimation for a high-dimensional distribution $\rho(x)$. Traditional approaches have focused on the limit of large number of data points $n$ and fixed dimension $d$. We analyze instead the regime where both the number $n$ of data points $y_i$ and their dimensionality $d$ grow with a fixed ratio $\alpha=(\log n)/d$. Our study reveals three distinct statistical regimes for the kernel-based estimate of the density $\hat \rho_h^{\mathcal {D}}(x)=\frac{1}{n h^d}\sum_{i=1}^n K\left(\frac{x-y_i}{h}\right)$, depending on the bandwidth $h$: a classical regime for large bandwidth where the Central Limit Theorem (CLT) holds, which is akin to the one found in traditional approaches. Below a certain value of the bandwidth, $h_{CLT}(\alpha)$, we find that the CLT breaks down. The statistics of $\hat\rho_h^{\mathcal {D}}(x)$ for a fixed $x$ drawn from $\rho(x)$ is given by a heavy-tailed distribution (an alpha-stable distribution). In particular below a value $h_G(\alpha)$, we find that $\hat\rho_h^{\mathcal {D}}(x)$ is governed by extreme value statistics: only a few points in the database matter and give the dominant contribution to the density estimator. We provide a detailed analysis for high-dimensional multivariate Gaussian data. We show that the optimal bandwidth threshold based on Kullback-Leibler divergence lies in the new statistical regime identified in this paper. As known by practitioners, when decreasing the bandwidth a Kernel-estimated estimated changes from a smooth curve to a collections of peaks centred on the data points. Our findings reveal that this general phenomenon is related to sharp transitions between phases characterized by different statistical properties, and offer new insights for Kernel density estimation in high-dimensional settings.

Gaussian RBF Centered Kernel Alignment (CKA) in the Large Bandwidth Limit

Self-representative kernel concept factorization

Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks

The Generalized Complex Kernel Affine Projection Algorithms

Canonical Correlation Analysis based on Hilbert-Schmidt Independence Criterion and Centered Kernel Target Alignment.

Rethinking Centered Kernel Alignment in Knowledge Distillation

Equivalence between representational similarity analysis, centered kernel alignment, and canonical correlations analysis

Rates of convergence of randomized Kaczmarz algorithms in Hilbert spaces

A new randomized Kaczmarz based kernel canonical correlation analysis algorithm with applications to information retrieval

The Kernel Conjugate Gradient Algorithms.

Data-Driven Linear Complexity Low-Rank Approximation of General Kernel Matrices: A Geometric Approach

Enhancing Kernel Flexibility Via Learning Asymmetric Locally-Adaptive Kernels.

Learning Kernels with Radiuses of Minimum Enclosing Balls.

Stable and Accurate Least Squares Radial Basis Function Approximations on Bounded Domains

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

A Convex Parametrization of a New Class of Universal Kernel Functions

Error Bounds for a Kernel-Based Constrained Optimal Smoothing Approximation

Invariance of Gaussian RKHSs under Koopman operators of stochastic differential equations with constant matrix coefficients

Regularization in kernel learning

Regularized Regression Problem in hyper-RKHS for Learning Kernels

Kernel Density Estimators in Large Dimensions