Abstract:Model selection is one of the fundamental problems in kernel-based algorithms, which is commonly done by minimizing an estimation of generalization error. The notion of stability and cross-validation (CV) error of learning machines consists of two widely used tools for analyzing the generalization performance. However, there are some disadvantages to both tools when applied for model selection: 1) the stability of learning machines is not practical due to the difficulty of the estimation of its specific value and 2) the CV-based estimate of generalization error usually has a relatively high variance, so it is prone to overfitting. To overcome these two limitations, we present a novel notion of kernel stability (KS) for deriving the generalization error bounds and variance bounds of CV and provide an effective approach to the application of KS for practical model selection. Unlike the existing notions of stability of the learning machine, KS is defined on the kernel matrix; hence, it can avoid the difficulty of the estimation of its value. We manifest the relationship between the KS and the popular uniform stability of the learning algorithm, and further propose several KS-based generalization error bounds and variance bounds of CV. By minimizing the proposed bounds, we present two novel KS-based criteria that can ensure good performance. Finally, we empirically analyze the performance of the proposed criteria on many benchmark data, which demonstrates that our KS-based criteria are sound and effective.

Combinatorial Kernel Matrix Model Selection Using Feature Distances

A Kernel Function Optimization And Selection Algorithm Based On Cost Function Maximization

Self-representative kernel concept factorization

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

Distance-Based Classifier Via the Kernel Trick.

Kernel Alignment for Unsupervised Feature Selection via Matrix Factorization

Framework for Choice of Kernels

Efficient Kernel Selection Via Spectral Analysis.

Feature Selection in a Kernel Space

Improved kernel direct Fisher discriminant analysis and face recognition

Matrix-Based Kernel Subspace Methods

Approximate Kernel Selection via Matrix Approximation

Model Selection for Support Vector Machines Based on Kernel Density Estimation

TAKES: a fast method to select features in the kernel space.

A binary feature selection framework in kernel spaces

Bi-criterion Optimization for Kernel Parameters Selection

Conditional mean embeddings and optimal feature selection via positive definite kernels

Optimizing the Kernel in the Empirical Feature Space

Optimizing the Data-Dependent Kernel under a Unified Kernel Optimization Framework

The Variant of Gaussian Kernel and Its Model Selection Method

Kernel Stability for Model Selection in Kernel-Based Algorithms.