Abstract:New optimization models and algorithms for online learning with Kernels (OLK) in classification, regression, and novelty detection are proposed in a reproducing Kernel Hilbert space. Unlike the stochastic gradient descent algorithm, called the naive online Reg minimization algorithm (NORMA), OLK algorithms are obtained by solving a constrained optimization problem based on the proposed models. By exploiting the techniques of the Lagrange dual problem like Vapnik's support vector machine (SVM), the solution of the optimization problem can be obtained iteratively and the iteration process is similar to that of the NORMA. This further strengthens the foundation of OLK and enriches the research area of SVM. We also apply the obtained OLK algorithms to problems in classification, regression, and novelty detection, including real time background substraction, to show their effectiveness. It is illustrated that, based on the experimental results of both classification and regression, the accuracy of OLK algorithms is comparable with traditional SVM-based algorithms, such as SVM and least square SVM (LS-SVM), and with the state-ofthe-art algorithms, such as Kernel recursive least square (KRLS) method and projectron method, while it is slightly higher than that of NORMA. On the other hand, the computational cost of the OLK algorithm is comparable with or slightly lower than existing online methods, such as above mentioned NORMA, KRLS, and projectron methods, but much lower than that of SVM-based algorithms. In addition, different from SVM and LS-SVM, it is possible for OLK algorithms to be applied to non-stationary problems. Also, the applicability of OLK in novelty detection is illustrated by simulation results.

Communication-Efficient Distributed Online Learning with Kernels

Online Kernel Learning with a Near Optimal Sparsity Bound

QC-ODKLA: Quantized and Communication-Censored Online Decentralized Kernel Learning via Linearized ADMM

An Online Multiple Kernel Parallelizable Learning Scheme

COKE: Communication-Censored Kernel Learning for Decentralized Non-parametric Learning.

COKE: Communication-Censored Decentralized Kernel Learning

Adaptation and Learning over Networks for Nonlinear System Modeling

Isolation Kernel: The X Factor in Efficient and Effective Large Scale Online Kernel Learning

Model-Based Online Learning with Kernels.

Distributed Online Optimization via Kernel Reproduced Gradient Descent

Large Scale Online Kernel Classification

Efficient Online Learning for Large-Scale Sparse Kernel Logistic Regression

Coke: Communication-Censored Kernel Learning Via Random Features.

Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filtering

Communication-Efficient Online Federated Learning Framework for Nonlinear Regression

LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression

A Communication-Efficient Distributed Algorithm for Kernel Principal Component Analysis.

High-performance Kernel Machines with Implicit Distributed Optimization and Randomization

Fast Bounded Online Gradient Descent Algorithms for Scalable Kernel-Based Online Learning

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient