Abstract:This paper explores numerical methods for solving a convex differentiable semi-infinite program. We introduce a primal-dual gradient method which performs three updates iteratively: a momentum gradient ascend step to update the constraint parameters, a momentum gradient ascend step to update the dual variables, and a gradient descend step to update the primal variables. Our approach also extends to scenarios where gradients and function values are accessible solely through stochastic oracles. This method extends the recent primal-dual methods, for example, Hamedani and Aybat (2021); Boob et al. (2022), for optimization with a finite number of constraints. We show the iteration complexity of the proposed method for finding an $\epsilon$-optimal solution under different convexity and concavity assumptions on the functions.

What problem does this paper attempt to address?

This paper aims to solve the Convex Semi - Infinite Optimization (CSIO) problem. Specifically, the authors explore numerical methods for solving a convex differentiable semi - infinite programming problem with an infinite number of constraints. This type of problem can be expressed as: \[ f^* := \min_{x \in X} f(x) \quad \text{s.t.} \quad g_i(x, y_i) \leq 0, \forall y_i \in Y_i, i = 1, \ldots, m. \] where: - $ f $ and $ g_i $ are real - valued continuous functions, - $ X\subset \mathbb{R}^p $ and $ Y_i\subset \mathbb{R}^q $ are convex closed sets. Since $ Y_i $ is not a finite set, the number of constraints is infinite, which makes the problem a Semi - Infinite Program (SIP). SIP has applications in many fields, such as robotics, statistics, machine learning, stochastic programming, robust optimization, Markov decision processes, inventory control, revenue management, and healthcare, etc. ### Main contributions of the paper 1. **Propose a new gradient method**: The authors introduce a primal - dual gradient method, which performs three update steps in each iteration: - A momentum gradient ascent step to update the constraint parameter $ y_i $. - A momentum gradient ascent step to update the dual variable $ \lambda $. - A gradient descent step to update the primal variable $ x $. 2. **Extension to stochastic scenarios**: This method is applicable not only to deterministic gradients and function values, but also to cases where gradients and function values can only be obtained through a stochastic oracle. 3. **Theoretical complexity analysis**: The authors analyze the number of iterations required to find an $ \epsilon $-optimal solution under different convexity and concavity assumptions. ### Problems solved The paper mainly solves the following problems: - How to effectively solve convex semi - infinite optimization problems with an infinite number of constraints. - Propose a new gradient method and analyze its convergence and complexity. - Extend the existing primal - dual method so that it can handle cases with an infinite number of constraints. Through these contributions, the paper provides a new and effective method for solving complex semi - infinite optimization problems.

Deterministic and Stochastic Accelerated Gradient Method for Convex Semi-Infinite Optimization

A Universal Accelerated Primal–Dual Method for Convex Optimization Problems

Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization

Accelerated Stochastic Subgradient Methods under Local Error Bound Condition

Gradient Methods with Dynamic Inexact Oracles

Accelerated gradient methods for nonconvex nonlinear and stochastic programming

Accelerated Primal-dual Scheme for a Class of Stochastic Nonconvex-concave Saddle Point Problems

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

Inertial Accelerated Primal-Dual Methods for Linear Equality Constrained Convex Optimization Problems.

Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition

Accelerated nonlinear primal-dual hybrid gradient methods with applications to supervised machine learning

Proximal-Proximal-Gradient Method

Decentralized Gradient-Free Methods for Stochastic Non-Smooth Non-Convex Optimization

Dual Descent Augmented Lagrangian Method and Alternating Direction Method of Multipliers

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Accelerating Primal-Dual Methods for Regularized Markov Decision Processes

On Stochastic Primal-Dual Hybrid Gradient Approach for Compositely Regularized Minimization.

On the Iteration Complexity Analysis of Stochastic Primal-Dual Hybrid Gradient Approach with High Probability

Accelerating Asynchronous Algorithms for Convex Optimization by Momentum Compensation

Asynchronous Decentralized Accelerated Stochastic Gradient Descent

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization