Abstract:Online kernel selection is a more complex problem compared with offline kernel selection, which intermixes training and selection at each round and requires a sublinear regret and low computational complexities. But existing online kernel selection approaches have at least linear time and space complexities at each round with respect to the number of rounds, or lack sublinear regret guarantees for an uncountably infinite number of candidate kernels. To address these issues, we propose a novel online kernel selection approach using tensor sketching, which has constant computational complexities at each round and enjoys a sublinear regret bound for an uncountably infinite number of candidate kernels. We represent the data using the tensor products and construct data sketches using the Taylor series and the Count Sketch matrices, which yields a sketched reproducing kernel Hilbert space (SRKHS). Then we update the optimal kernels and the hypotheses using online gradient descent in SRKHS. We prove that the kernel corresponding to SRKHS satisfies the reproducing property, the hypotheses in SRKHS are convex with respect to the kernel parameter, and the proposed online kernel selection approach in SRKHS enjoys a regret bound of order $O(\sqrtT )$ for an uncountably infinite number of candidate kernels, which is optimal for a convex loss function, where T is the number of rounds. By the fast Fourier transform, the hypotheses in SRKHS can be computed in a quasilinear time complexity and a logarithmic space complexity with respect to the sketch size at each round, where the sketch size is a constant. Experimental results demonstrate that our online kernel selection approach is more accurate and efficient for online kernel learning on high dimension data.

Dependent Online Kernel Learning with Constant Number of Random Fourier Features

On the Sample Complexity of Random Fourier Features for Online Learning: How Many Random Fourier Features Do We Need?

Online Kernel Learning with a Near Optimal Sparsity Bound

Online Kernel Learning with Nearly Constant Support Vectors

Learning Adaptive Random Features

Fast and Strong Convergence of Online Learning Algorithms.

Online Learning in Reproducing Kernel Hilbert Space with Non-IID Data

FFTRL: A Sparse Online Kernel Classification Algorithm for Large Scale Data

Online Learning with Samples Drawn from Non-identical Distributions

Large Scale Online Kernel Classification

Convergence of Unregularized Online Learning Algorithms

Distributed Learning With Dependent Samples

Convergence Conditions of Online Regularized Statistical Learning in Reproducing Kernel Hilbert Space With Non-Stationary Data

Isolation Kernel: The X Factor in Efficient and Effective Large Scale Online Kernel Learning

Random Fourier Features Kernel Recursive Generalized Loss for Online Time Series Prediction

Deep Kernel Learning via Random Fourier Features

Learning Random Fourier Features by Hybrid Constrained Optimization

Random Fourier Features Based Kernel Recursive Least Logarithmic Hyperbolic Cosine Algorithm.

Random Inverse Problems Over Graphs: Decentralized Online Learning

Distributed Online Optimization via Kernel Reproduced Gradient Descent

Online Kernel Selection Via Tensor Sketching