Abstract:Positive definite kernels and their associated Reproducing Kernel Hilbert Spaces provide a mathematically compelling and practically competitive framework for learning from data. In this paper we take the approximation theory point of view to explore various aspects of smooth kernels related to their inferential properties. We analyze eigenvalue decay of kernels operators and matrices, properties of eigenfunctions/eigenvectors and "Fourier" coefficients of functions in the kernel space restricted to a discrete set of data points. We also investigate the fitting capacity of kernels, giving explicit bounds on the fat shattering dimension of the balls in Reproducing Kernel Hilbert spaces. Interestingly, the same properties that make kernels very effective approximators for functions in their "native" kernel space, also limit their capacity to represent arbitrary functions. We discuss various implications, including those for gradient descent type methods. It is important to note that most of our bounds are measure independent. Moreover, at least in moderate dimension, the bounds for eigenvalues are much tighter than the bounds which can be obtained from the usual matrix concentration results. For example, we see that the eigenvalues of kernel matrices show nearly exponential decay with constants depending only on the kernel and the domain. We call this "approximation beats concentration" phenomenon as even when the data are sampled from a probability distribution, some of their aspects are better understood in terms of approximation theory.

Kernel Based Approximation in Sobolev Spaces with Radial Basis Functions

Extending error bounds for radial basis function interpolation to measuring the error in higher order Sobolev norms

Interpolation and Best Approximation for Spherical Radial Basis Function Networks

Error Bounds for a Kernel-Based Constrained Optimal Smoothing Approximation

Regular Families of Kernels for Nonlinear Approximation

Approximation of Bivariate Functions by Generalized Wendland Radial Basis Functions

Stable and Accurate Least Squares Radial Basis Function Approximations on Bounded Domains

Numerical Linear Approximation Involving Radial Basis Functions

On the Approximation of Kernel functions

Approximation beats concentration? An approximation view on inference with smooth radial kernels

Local Multilevel Scattered Data Interpolation

A new type of radial basis functions for problems governed by partial differential equations

Smoothing Splines Using Compactly Supported, Positive Definite, Radial Basis Functions

On Spectral Distribution of Kernel Matrices Related to Radial Basis Functions

Scattered Data Interpolation on Embedded Submanifolds with Restricted Positive Definite Kernels: Sobolev Error Estimates

Radial polynomials as alternatives to smooth radial basis functions and their applications

Local Error Estimates for Radial Basis Function Interpolation of Scattered Data

L2 Error Estimates of Unsymmetric RBF Collocation for Second Order Elliptic Boundary Value Problems

Scattered Data Interpolation and Approximation with Truncated Exponential Radial Basis Function

Approximation on Variable Exponent Spaces by Linear Integral Operators