Abstract:The doubly stochastic functional gradient descent algorithm (DSG) that is memory friendly and computationally efficient can effectively scale up kernel methods. However, in solving the highly ill-conditioned large-scale nonlinear machine learning problem, the convergence speed of DSG is quite slow. This is because the condition number of the Hessian matrix of this problem is quite large, which will make stochastic gradient methods converge very slowly. Fortunately, gradient preconditioning is a well-established technique in optimization aiming to reduce the condition number. Therefore, we propose a preconditioned doubly stochastic functional gradient descent algorithm (P-DSG) by combining DSG with gradient preconditioning. P-DSG first uses the gradient preconditioning to adaptively scale the individual components of the estimated functional gradient obtained by DSG, and then utilizes the preconditioned functional gradient as the descent direction in each iteration. Theoretically, an appropriate preconditioner is always the inverse of the Hessian matrix at the optimum, which is not easy to get due to its high computation cost. Therefore, we first choose an empirical covariance matrix of random Fourier features to approximate the Hessian matrix, and then perform a low-rank approximation to the empirical covariance matrix. P-DSG has a fast convergence rate O(1/t)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$\mathcal {O}(1/t)$\end{document} and produces a smaller constant factor in the boundary than that of DSG while remains O(t)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$\mathcal {O}(t)$\end{document} memory friendly and O(td)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$\mathcal {O}(td)$\end{document} computationally efficient. Finally, we test the performance of P-DSG on the kernel ridge regression, kernel support vector machines, and kernel logistic regression, respectively. The experimental results show that P-DSG speeds up convergence and achieves better performance.

Adaptive Preconditioned Gradient Descent with Energy

Dynamic behavior for a gradient algorithm with energy and momentum

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix

Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization

Preconditioning for Accelerated Gradient Descent Optimization and Regularization

ELRA: Exponential learning rate adaption gradient descent optimization method

Stochastic Gradient Descent with Preconditioned Polyak Step-size

A gradient descent akin method for constrained optimization: algorithms and applications

An Energy-Based Self-Adaptive Learning Rate for Stochastic Gradient Descent: Enhancing Unconstrained Optimization with VAV method

Evolutionary Gradient Descent for Non-convex Optimization

AdaX: Adaptive Gradient Descent with Exponential Long Term Memory

Accelerated Proximal Coordinate Gradient Methods

On The Unified Design Of Accelerated Gradient Descent

Faster doubly stochastic functional gradient by gradient preconditioning for scalable kernel methods

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

A New Adaptive Gradient Method with Gradient Decomposition

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

Optimal Adaptive and Accelerated Stochastic Gradient Descent

AGDA+: Proximal Alternating Gradient Descent Ascent Method With a Nonmonotone Adaptive Step-Size Search For Nonconvex Minimax Problems