Abstract:Classical convergence analyses for optimization algorithms rely on the widely-adopted uniform smoothness assumption. However, recent experimental studies have demonstrated that many machine learning problems exhibit non-uniform smoothness, meaning the smoothness factor is a function of the model parameter instead of a universal constant. In particular, it has been observed that the smoothness grows with respect to the gradient norm along the training trajectory. Motivated by this phenomenon, the recently introduced $(L_0, L_1)$-smoothness is a more general notion, compared to traditional $L$-smoothness, that captures such positive relationship between smoothness and gradient norm. Under this type of non-uniform smoothness, existing literature has designed stochastic first-order algorithms by utilizing gradient clipping techniques to obtain the optimal $\mathcal{O}(\epsilon^{-3})$ sample complexity for finding an $\epsilon$-approximate first-order stationary solution. Nevertheless, the studies of quasi-Newton methods are still lacking. Considering higher accuracy and more robustness for quasi-Newton methods, in this paper we propose a fast stochastic quasi-Newton method when there exists non-uniformity in smoothness. Leveraging gradient clipping and variance reduction, our algorithm can achieve the best-known $\mathcal{O}(\epsilon^{-3})$ sample complexity and enjoys convergence speedup with simple hyperparameter tuning. Our numerical experiments show that our proposed algorithm outperforms the state-of-the-art approaches.

A Nonmonotone Smoothing Newton Method for System of Nonlinear Inequalities Based on a New Smoothing Function

A Smoothing-Type Algorithm for Solving System of Inequalities

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

An Improved Smoothing Technique-Based Control Vector Parameterization Method for Optimal Control Problems with Inequality Path Constraints

On the Finite Convergence of Newton-type Methods for P 0 Affine Variational Inequalities

Improved Smoothing Newton Methods for P0 Nonlinear Complementarity Problems

A Globalized Inexact Semismooth Newton Method for Nonsmooth Fixed-point Equations involving Variational Inequalities

Nonlinearly preconditioned semismooth Newton algorithms for nonlinear nonsmooth systems

Global Linear and Quadratic One-Step Smoothing Newton Method for Vertical Linear Complementarity Problems

Superlinear／Quadratic One-step Smoothing Newton Methodf or P0-NCP

Extended Newton Methods for Conic Inequalities: Approximate Solutions and the Extended Smale Α-Theory

A HYBRID SMOOTHING-NONSMOOTH NEWTON-TYPE ALGORITHM YIELDING AN EXACT SOLUTION OF THE

Superlinear/quadratic one-step smoothing Newton method for P 0 -NCP without strict complementarity

A Jacobian smoothing inexact Newton method for solving the nonlinear complementary problem

Implementable Semismooth* Newton Methods for Generalized Equations are G-Semismooth Newton Methods

Superlinear/Quadratic One–step Smoothing Newton Method for P 0–NCP

A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness

Calibrating Least Squares Semidefinite Programming with Equality and Inequality Constraints

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization

Global Linear and Quadratic One-step Smoothing Newton Method for P 0 -LCP

A QP-FREE AND SUPERLINEARLY CONVERGENT ALGORITHM FOR INEQUALITY CONSTRAINEDOPTIMIZATIONS