Abstract:In this paper we revisit the kernel density estimation problem: given a kernel $K(x, y)$ and a dataset of $n$ points in high dimensional Euclidean space, prepare a data structure that can quickly output, given a query $q$, a ($1+ \epsilon$)-approximation to $\mu:=\frac{1}{\vert P\vert}\sum\nolimits_{p\in P}K(p, q)$. First, we give a single data structure based on classical near neighbor search techniques that improves upon or essentially matches the query time and space complexity for all radial kernels considered in the literature so far. We then show how to improve both the query complexity and runtime by using recent advances in data-dependent near neighbor search. We achieve our results by giving an new implementation of the natural importance sampling scheme. Unlike previous approaches, our algorithm first samples the dataset uniformly (considering a geometric sequence of sampling rates), and then uses existing approximate near neighbor search techniques on the resulting smaller dataset to retrieve the sampled points that lie at an appropriate distance from the query. We show that the resulting sampled dataset has strong geometric structure, making approximate near neighbor search return the required samples much more efficiently than for worst case datasets of the same size. As an example application, we show that this approach yields a data structure that achieves query time and space complexity for the Gaussian kernel. Our data dependent approach achieves query time and space for the Gaussian kernel. The data dependent analysis relies on new techniques for tracking the geometric structure of the input datasets in a recursive hashing process that we hope will be of interest in other applications in near neighbor search.

On the Approximation of Kernel functions

Approximation in Hilbert spaces of the Gaussian and related analytic kernels

Approximation beats concentration? An approximation view on inference with smooth radial kernels

A new kernel method for the uniform approximation in reproducing kernel Hilbert spaces

Almost-sure quasi-optimal approximation in reproducing kernel Hilbert spaces

Regular Families of Kernels for Nonlinear Approximation

Multi- and Infinite-variate Integration and $L^2$-Approximation on Hilbert Spaces with Gaussian Kernels

Error Bounds for a Kernel-Based Constrained Optimal Smoothing Approximation

Approximation of RKHS Functionals by Neural Networks

Error bounds for kernel-based approximations of the Koopman operator

Uniform approximation of common Gaussian process kernels using equispaced Fourier grids

Approximating Reproducing Kernel Hilbert Space Functions by Bernstein Operators

Data-Driven Linear Complexity Low-Rank Approximation of General Kernel Matrices: A Geometric Approach

Approximation by non-symmetric networks for cross-domain learning

Approximate Kernel Selection via Matrix Approximation

Learning Theory Viewpoint of Approximation by Positive Linear Operators

Polyharmonic and Related Kernels on Manifolds: Interpolation and Approximation

On application of Gaussian kernels and Laplace functions combined with Kolmogorov's theorem for approximation of functions of several variables

Kernel Density Estimation through Density Constrained Near Neighbor Search

Kernel Based Approximation in Sobolev Spaces with Radial Basis Functions