Abstract:The doubly stochastic functional gradient descent algorithm (DSG) that is memory friendly and computationally efficient can effectively scale up kernel methods. However, in solving the highly ill-conditioned large-scale nonlinear machine learning problem, the convergence speed of DSG is quite slow. This is because the condition number of the Hessian matrix of this problem is quite large, which will make stochastic gradient methods converge very slowly. Fortunately, gradient preconditioning is a well-established technique in optimization aiming to reduce the condition number. Therefore, we propose a preconditioned doubly stochastic functional gradient descent algorithm (P-DSG) by combining DSG with gradient preconditioning. P-DSG first uses the gradient preconditioning to adaptively scale the individual components of the estimated functional gradient obtained by DSG, and then utilizes the preconditioned functional gradient as the descent direction in each iteration. Theoretically, an appropriate preconditioner is always the inverse of the Hessian matrix at the optimum, which is not easy to get due to its high computation cost. Therefore, we first choose an empirical covariance matrix of random Fourier features to approximate the Hessian matrix, and then perform a low-rank approximation to the empirical covariance matrix. P-DSG has a fast convergence rate O(1/t)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$\mathcal {O}(1/t)$\end{document} and produces a smaller constant factor in the boundary than that of DSG while remains O(t)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$\mathcal {O}(t)$\end{document} memory friendly and O(td)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$\mathcal {O}(td)$\end{document} computationally efficient. Finally, we test the performance of P-DSG on the kernel ridge regression, kernel support vector machines, and kernel logistic regression, respectively. The experimental results show that P-DSG speeds up convergence and achieves better performance.

Fast Gaussian Kernel Learning for Classification Tasks Based on Specially Structured Global Optimization

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Parallel Stochastic Optimization Framework for Large-Scale Non-Convex Stochastic Problems

Large Scale Online Kernel Classification

Efficient sparse generalized multiple kernel learning

Fast Multi-View Segment Graph Kernel for Object Classification

Kernel Multigrid: Accelerate Back-fitting via Sparse Gaussian Process Regression

Global Optimization of Gaussian processes

Learning Local Graph from Multiple Kernels

On Valid Optimal Assignment Kernels and Applications to Graph Classification

Faster doubly stochastic functional gradient by gradient preconditioning for scalable kernel methods

Kernelized discriminative–collaborative representation-based approach for pattern classification

Fast Bounded Online Gradient Descent Algorithms for Scalable Kernel-Based Online Learning

Speeding up the binary Gaussian process classification

Batch Sequential Adaptive Designs for Global Optimization

The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines

Online Kernel Dictionary Learning for Object Recognition

Learning sparse kernel classifiers in the primal

Multiple graph kernel learning based on GMDH-type neural network

Efficient Two-Stage Gaussian Process Regression Via Automatic Kernel Search and Subsampling

An efficient constrained global optimization algorithm with a clustering-assisted multiobjective infill criterion using Gaussian process regression for expensive problems