Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 1374-1401, June 2024. A popular approach to minimizing a finite sum of smooth convex functions is stochastic gradient descent (SGD) and its variants. Fundamental research questions associated with SGD include (i) how to find a lower bound on the number of times that the gradient oracle of each individual function must be assessed in order to find an [math]-minimizer of the overall objective; (ii) how to design algorithms which guarantee finding an [math]-minimizer of the overall objective in expectation no more than a certain number of times (in terms of [math]) that the gradient oracle of each function needs to be assessed (i.e., upper bound). If these two bounds are at the same order of magnitude, then the algorithms may be called optimal. Most existing results along this line of research typically assume that the functions in the objective share the same condition number. In this paper, the first model we study is the problem of minimizing the sum of finitely many strongly convex functions whose condition numbers are all different. We propose an SGD-based method for this model and show that it is optimal in gradient computations, up to a logarithmic factor. We then consider a constrained separate block optimization model and present lower and upper bounds for its gradient computation complexity. Next, we propose solving the Fenchel dual of the constrained block optimization model via generalized SSNM, which we introduce earlier, and show that it yields a lower iteration complexity than solving the original model by the ADMM-type approach. Finally, we extend the analysis to the general composite convex optimization model and obtain gradient-computation complexity results under certain conditions.

Optimality Conditions for Minimizing the Difference of Nonconvex Vector-Valued Mappings

Optimality conditions at infinity for nonsmooth minimax programming

Optimality Conditions of Strict Minimality in Optimization Problems under Inclusion Constraints.

Optimality conditions and sensitivity analysis in parametric convex minimax programming

Second-order optimality conditions for general nonconvex optimization problems and variational analysis of disjunctive systems

Optimality and duality in nonsmooth multiobjective fractional programming problem with constraints

Variational Inequalities for Set-Valued Vector Fields on Riemannian Manifolds: Convexity of the Solution Set and the Proximal Point Algorithm.

Optimization Condition and Algorithm of Optimization with Convertible Nonconvex Function

Optimality conditions for the nonlinear programming problems on Riemannian manifolds

Efficiency Conditions for Nonsmooth Vector Equilibrium Problems with Constraints via Generalized Subdifferentials

The Continuity and Convexity of a Nonlinear Scalarization Function with Applications in Set Optimization Problems Involving a Partial Order Relation

First-order Optimality Conditions for Two Classes of Generalized Nonsmooth Semi-Infinite Optimization

Vector optimization problems in near vector space

A Gradient Complexity Analysis for Minimizing the Sum of Strongly Convex Functions with Varying Condition Numbers

Lagrange Multiplier Local Necessary and Global Sufficiency Criteria for Some Non-Convex Programming Problems

Some optimality conditions of set-valued optimization problems in locally convex topological vector spaces

The minimizer of the sum of two strongly convex functions

Optimality Conditions and Duality for a Class of Nonlinear Fractional Programming Problems

Dual Kuhn-Tucker necessary conditions for strict minima of order two for nonsmooth vector optimization problems with inequality and equality constraints

Optimality conditions and subdifferential calculus for infinite sums of functions

Non-subdifferentiability optimality and mean value theorems via new relative subdifferentials