Abstract:In this paper, we focus on improving the performance of the Nyström based kernel SVM. Although the Nyström approximation has been studied extensively and its application to kernel classification has been exhibited in several studies, there still exists a potentially large gap between the performance of classifier learned with the Nyström approximation and that learned with the original kernel. In this work, we make novel contributions to bridge the gap without increasing the training costs too much by proposing a refined Nyström based kernel classifier. We adopt a two-step approach that in the first step we learn a sufficiently good dual solution and in the second step we use the obtained dual solution to construct a new set of bases for the Nyström approximation to re-train a refined classifier. Our approach towards learning a good dual solution is based on a sparse-regularized dual formulation with the Nyström approximation, which can be solved with the same time complexity as solving the standard formulation. We justify our approach by establishing a theoretical guarantee on the error of the learned dual solution in the first step with respect to the optimal dual solution under appropriate conditions. The experimental results demonstrate that (i) the obtained dual solution by our approach in the first step is closer to the optimal solution and yields improved prediction performance; and (ii) the second step using the obtained dual solution to re-train the model further improves the performance.

Online Kernel Learning with Nearly Constant Support Vectors

Online Kernel Learning with a Near Optimal Sparsity Bound

On-line support vector regression with multiple samples

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Dependent Online Kernel Learning with Constant Number of Random Fourier Features

Large Scale Online Kernel Classification

Incremental batch learning with support vector machines

Mini-batch Quasi-Newton Optimization for Large Scale Linear Support Vector Regression

On the Sample Complexity of Random Fourier Features for Online Learning: How Many Random Fourier Features Do We Need?

On-line Support Vector Machine Training Algorithm and Its Application

Nonlinear Online Learning with Adaptive Nyström Approximation

Efficient Online Learning for Large-Scale Sparse Kernel Logistic Regression

Fast Bounded Online Gradient Descent Algorithms for Scalable Kernel-Based Online Learning

Fast and Accurate Refined Nyström-Based Kernel SVM

Non-uniform Nyström approximation for sparse kernel regression: Theoretical analysis and experimental evaluation

Improved Bounds for the Nyström Method with Application to Kernel Classification

Budget Online Multiple Kernel Learning

Local least squares support vector regression with application to online modeling for batch processes

Learning a Coupled Linearized Method in Online Setting.

Limited Memory Online Gradient Descent for Kernelized Pairwise Learning with Dynamic Averaging

A Semismooth-Newton's-Method-Based Linearization and Approximation Approach for Kernel Support Vector Machines