Abstract:This paper generalizes the two-stage kernel learning framework, illustrates its utility for kernel learning and out-of-sample extensions, and proves {asymptotic} convergence results for the introduced kernel learning model. Algorithmically, we extend target alignment by hyper-kernels in the two-stage kernel learning framework. The associated kernel learning task is formulated as a regression problem in a hyper-reproducing kernel Hilbert space (hyper-RKHS), i.e., learning on the space of kernels itself. To solve this problem, we present two regression models with bivariate forms in this space, including kernel ridge regression (KRR) and support vector regression (SVR) in the hyper-RKHS. By doing so, it provides significant model flexibility for kernel learning with outstanding performance in real-world applications. Specifically, our kernel learning framework is general, that is, the learned underlying kernel can be positive definite or indefinite, which adapts to various requirements in kernel learning. Theoretically, we study the convergence behavior of these learning algorithms in the hyper-RKHS and derive the learning rates. Different from the traditional approximation analysis in RKHS, our analyses need to consider the non-trivial independence of pairwise samples and the characterisation of hyper-RKHS. To the best of our knowledge, this is the first work in learning theory to study the approximation performance of regularized regression problem in hyper-RKHS.

Consistency and Error Analysis of Prior-Knowledge-Based Kernel Regression

Incorporating Prior Knowledge into KernelBased Regression

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

Robust Regularized Kernel Regression.

Towards a Unified Analysis of Kernel-based Methods Under Covariate Shift

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

A General Kernelization Framework for Learning Algorithms Based on Kernel PCA.

Regularized Regression Problem in hyper-RKHS for Learning Kernels

Iterative Kernel Regression with Preconditioning

Low-rank kernel regression with preserved locality for multi-class analysis

Probabilistic Analysis of Kernel Principal Components

Multivariate Universal Local Linear Kernel Estimators in Nonparametric Regression: Uniform Consistency

Analysis of Knowledge Transfer in Kernel Regime

Kernel Partial Least Squares is Universally Consistent

An efficient multiple kernel computation method for regression analysis of economic data

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Model selection of polynomial kernel regression

Study on Kernel Partial Least Squares Based Key Indicator Prediction.

Error Analysis of Kernel/GP Methods for Nonlinear and Parametric PDEs

Robust kernel ensemble regression in diversified kernel space with shared parameters