Abstract:Classical convergence analyses for optimization algorithms rely on the widely-adopted uniform smoothness assumption. However, recent experimental studies have demonstrated that many machine learning problems exhibit non-uniform smoothness, meaning the smoothness factor is a function of the model parameter instead of a universal constant. In particular, it has been observed that the smoothness grows with respect to the gradient norm along the training trajectory. Motivated by this phenomenon, the recently introduced $(L_0, L_1)$-smoothness is a more general notion, compared to traditional $L$-smoothness, that captures such positive relationship between smoothness and gradient norm. Under this type of non-uniform smoothness, existing literature has designed stochastic first-order algorithms by utilizing gradient clipping techniques to obtain the optimal $\mathcal{O}(\epsilon^{-3})$ sample complexity for finding an $\epsilon$-approximate first-order stationary solution. Nevertheless, the studies of quasi-Newton methods are still lacking. Considering higher accuracy and more robustness for quasi-Newton methods, in this paper we propose a fast stochastic quasi-Newton method when there exists non-uniformity in smoothness. Leveraging gradient clipping and variance reduction, our algorithm can achieve the best-known $\mathcal{O}(\epsilon^{-3})$ sample complexity and enjoys convergence speedup with simple hyperparameter tuning. Our numerical experiments show that our proposed algorithm outperforms the state-of-the-art approaches.

A trust region-type normal map-based semismooth Newton method for nonsmooth nonconvex composite optimization

A TRUST-REGION METHOD FOR NONSMOOTH NONCONVEX OPTIMIZATION

A Trust Region-Type Method for Solving Monotone Variational Inequality

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

On the local convergence of the semismooth Newton method for composite optimization

A nonmonotone trust region method for unconstrained optimization

A globally convergent proximal Newton-type method in nonsmooth convex optimization

An interior-point trust-region method for nonsmooth regularized bound-constrained optimization

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

A Trust Region Proximal Gradient Method for Nonlinear Multi-objective Optimization Problems

A nonmonotone adaptive trust region method for unconstrained optimization based on conic model

A new trust region algorithm for nonsmooth convex minimization

Beyond Nonconvexity: A Universal Trust-Region Method with New Analyses

A projected semismooth Newton method for a class of nonconvex composite programs with strong prox-regularity

A new nonmonotone trust-region method of conic model for solving unconstrained optimization

An adaptive simple model trust region algorithm based on new weak secant equations

An Improved Non-Monotonic Adaptive Trust Region Algorithm for Unconstrained Optimization

Generalized Newton Method with Positive Definite Regularization for Nonsmooth Optimization Problems with Nonisolated Solutions

A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness