Abstract:Stochastic gradient methods for minimizing nonconvex composite objective functions typically rely on the Lipschitz smoothness of the differentiable part, but this assumption fails in many important problem classes, leading to instability of the algorithms in both theory and practice. To address this, we propose a family of stochastic Bregman proximal gradient (SBPG) methods that only require smooth adaptivity. SBPG replaces the quadratic approximation in SGD with a Bregman proximity measure, offering a better approximation model that handles non-Lipschitz gradients in nonconvex objectives. We establish the convergence properties of vanilla SBPG and show it achieves optimal sample complexity in the nonconvex setting. Experimental results on quadratic inverse problems demonstrate SBPG's robustness in terms of stepsize selection and sensitivity to the initial point. Furthermore, we introduce a momentum-based variant, MSBPG, which enhances convergence by relaxing the mini-batch size requirement while preserving the optimal oracle complexity. We apply a polynomial kernel function based MBPG to the loss function with polynomial growth. Experimental results on benchmark datasets confirm the effectiveness and robustness of MSBPG. Given its negligible additional computational cost compared to SGD in large-scale optimization, MSBPG shows promise as a universal optimizer for future applications.

Parameter-free proximal bundle methods with adaptive stepsizes for hybrid convex composite optimization problems

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

An Adaptive Proximal Bundle Method with Inexact Oracles for a Class of Nonconvex and Nonsmooth Composite Optimization

Proximal bundle methods for hybrid weakly convex composite optimization problems

A proximal bundle variant with optimal iteration-complexity for a large range of prox stepsizes

Complexity and numerical experiments of a new adaptive generic proximal bundle method

Universal subgradient and proximal bundle methods for convex and strongly convex hybrid composite optimization

An Implementable Bundle Method for Nonsmooth Convex Optimization

Adaptive Proximal Average Approximation for Composite Convex Minimization.

A Proximal-Gradient Method for Constrained Optimization

A single cut proximal bundle method for stochastic convex composite optimization

Decentralized Inexact Proximal Gradient Method With Network-Independent Stepsizes for Convex Composite Optimization

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems

An Adaptive Proximal ADMM for Nonconvex Linearly-Constrained Composite Programs

Primal-dual proximal bundle and conditional gradient methods for convex problems

A Class of Infeasible Proximal Bundle Methods for Nonsmooth Nonconvex Multi-Objective Optimization Problems

A Proximal Bundle Method with Exact Penalty Technique and Bundle Modification Strategy for Nonconvex Nonsmooth Constrained Optimization

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

Barzilai-Borwein Proximal Gradient Methods for Multiobjective Composite Optimization Problems with Improved Linear Convergence

A Bundle Proximal Method for Solving General Mixed Variational Inequalities

An Inexact Preconditioned Zeroth-order Proximal Method for Composite Optimization