Abstract:This paper generalizes the two-stage kernel learning framework, illustrates its utility for kernel learning and out-of-sample extensions, and proves {asymptotic} convergence results for the introduced kernel learning model. Algorithmically, we extend target alignment by hyper-kernels in the two-stage kernel learning framework. The associated kernel learning task is formulated as a regression problem in a hyper-reproducing kernel Hilbert space (hyper-RKHS), i.e., learning on the space of kernels itself. To solve this problem, we present two regression models with bivariate forms in this space, including kernel ridge regression (KRR) and support vector regression (SVR) in the hyper-RKHS. By doing so, it provides significant model flexibility for kernel learning with outstanding performance in real-world applications. Specifically, our kernel learning framework is general, that is, the learned underlying kernel can be positive definite or indefinite, which adapts to various requirements in kernel learning. Theoretically, we study the convergence behavior of these learning algorithms in the hyper-RKHS and derive the learning rates. Different from the traditional approximation analysis in RKHS, our analyses need to consider the non-trivial independence of pairwise samples and the characterisation of hyper-RKHS. To the best of our knowledge, this is the first work in learning theory to study the approximation performance of regularized regression problem in hyper-RKHS.

Learning Rates of Least-Square Regularized Regression with Polynomial Kernels

Learning Rates for Regularized Classifiers Using Multivariate Polynomial Kernels

Full length article: Approximation by multivariate Bernstein-Durrmeyer operators and learning rates of least-squares regularized regression with multivariate polynomial kernels

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Optimal rate of the regularized regression learning algorithm

Learning rates for regularized least squares ranking algorithm

Analysis of Regularized Least-Squares in Reproducing Kernel Kreĭn Spaces

Least Square Regularized Regression in Sum Space.

Optimal Rates for Coefficient-Based Regularized Regression

Learning Rates for -Regularized Kernel Classifiers

Learning Theory Estimates for Coefficient-Based Regularized Regression

Optimal learning rates for Kernel Conjugate Gradient regression

Learning with coefficient-based regularization and ℓ 1 −penalty

Learning Rates of Regularized Regression on the Unit Sphere

Regularization in kernel learning

CLASSIFICATION WITH POLYNOMIAL KERNELS AND $L^1-$Coefficient REGULARIZATION

Optimal Learning Rates for Distribution Regression

Regularized Regression Problem in hyper-RKHS for Learning Kernels

Learning Theory Viewpoint of Approximation by Positive Linear Operators

Analysis of Regularized Nyström Subsampling for Regression Functions of Low Smoothness

Model selection of polynomial kernel regression