Abstract:The primary goal of this paper is to provide an efficient solution algorithm based on the augmented Lagrangian framework for optimization problems with a stochastic objective function and deterministic constraints. Our main contribution is combining the augmented Lagrangian framework with adaptive sampling, resulting in an efficient optimization methodology validated with practical examples. To achieve the presented efficiency, we consider inexact solutions for the augmented Lagrangian subproblems, and through an adaptive sampling mechanism, we control the variance in the gradient estimates. Furthermore, we analyze the theoretical performance of the proposed scheme by showing equivalence to a gradient descent algorithm on a Moreau envelope function, and we prove sublinear convergence for convex objectives and linear convergence for strongly convex objectives with affine equality constraints. The worst-case sample complexity of the resulting algorithm, for an arbitrary choice of penalty parameter in the augmented Lagrangian function, is $O(\epsilon^{-3-\delta})$, where $\epsilon > 0$ is the expected error of the solution and $\delta > 0$ is a user-defined parameter. If the penalty parameter is chosen to be $O(\epsilon^{-1})$, we demonstrate that the result can be improved to $O(\epsilon^{-2})$, which is competitive with the other methods employed in the literature. Moreover, if the objective function is strongly convex with affine equality constraints, we obtain $O(\epsilon^{-1}\log(1/\epsilon))$ complexity. Finally, we empirically verify the performance of our adaptive sampling augmented Lagrangian framework in machine learning optimization and engineering design problems, including topology optimization of a heat sink with environmental uncertainty.

Solving Stochastic Optimization with Expectation Constraints Efficiently by a Stochastic Augmented Lagrangian-Type Algorithm

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Algorithms for stochastic optimization with functional or expectation constraints

Stochastic Approximation for Expectation Objective and Expectation Inequality-Constrained Nonconvex Optimization

O(log T) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions

Stochastic Approximation Proximal Subgradient Method for Stochastic Convex-Concave Minimax Optimization

An Adaptive Sampling Augmented Lagrangian Method for Stochastic Optimization with Deterministic Constraints

Stochastic Optimization with Constraints: A Non-asymptotic Instance-Dependent Analysis

Sequential Quadratic Optimization for Stochastic Optimization with Deterministic Nonlinear Inequality and Equality Constraints

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

Robust Stochastic Approximation Approach to Stochastic Programming

Optimal Primal-Dual Algorithm with Last iterate Convergence Guarantees for Stochastic Convex Optimization Problems

Stochastic subgradient for composite optimization with functional constraints

An augmented Lagrangian trust-region method with inexact gradient evaluations to accelerate constrained optimization problems using model hyperreduction

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

Inequality constrained stochastic nonlinear optimization via active-set sequential quadratic programming

Near Optimal Stochastic Algorithms for Finite-Sum Unbalanced Convex-Concave Minimax Optimization

Optimal Stochastic Strongly Convex Optimization with a Logarithmic Number of Projections

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the Convergence Rate

A Stochastic Inexact Sequential Quadratic Optimization Algorithm for Nonlinear Equality-Constrained Optimization

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.