Abstract:The primary goal of this paper is to provide an efficient solution algorithm based on the augmented Lagrangian framework for optimization problems with a stochastic objective function and deterministic constraints. Our main contribution is combining the augmented Lagrangian framework with adaptive sampling, resulting in an efficient optimization methodology validated with practical examples. To achieve the presented efficiency, we consider inexact solutions for the augmented Lagrangian subproblems, and through an adaptive sampling mechanism, we control the variance in the gradient estimates. Furthermore, we analyze the theoretical performance of the proposed scheme by showing equivalence to a gradient descent algorithm on a Moreau envelope function, and we prove sublinear convergence for convex objectives and linear convergence for strongly convex objectives with affine equality constraints. The worst-case sample complexity of the resulting algorithm, for an arbitrary choice of penalty parameter in the augmented Lagrangian function, is $O(\epsilon^{-3-\delta})$, where $\epsilon > 0$ is the expected error of the solution and $\delta > 0$ is a user-defined parameter. If the penalty parameter is chosen to be $O(\epsilon^{-1})$, we demonstrate that the result can be improved to $O(\epsilon^{-2})$, which is competitive with the other methods employed in the literature. Moreover, if the objective function is strongly convex with affine equality constraints, we obtain $O(\epsilon^{-1}\log(1/\epsilon))$ complexity. Finally, we empirically verify the performance of our adaptive sampling augmented Lagrangian framework in machine learning optimization and engineering design problems, including topology optimization of a heat sink with environmental uncertainty.

Learn-and-Adapt Stochastic Dual Gradients for Network Resource Allocation

Stochastic Averaging for Constrained Optimization With Application to Online Resource Allocation

Optimal Regularized Online Allocation by Adaptive Re-Solving

Distributed Adaptive Gradient Algorithm with Gradient Tracking for Stochastic Non-Convex Optimization

A probabilistic framework with the gradient-based method for multi-objective land use optimization

An Adaptive Sampling Augmented Lagrangian Method for Stochastic Optimization with Deterministic Constraints

Distributed Dual Gradient Tracking for Resource Allocation in Unbalanced Networks

Stochastic Gradient Descent with Adaptive Data

A Data-Driven Approach to Stochastic Network Optimization

Online Optimization for Network Resource Allocation and Comparison with Reinforcement Learning Techniques

DSA: Decentralized Double Stochastic Averaging Gradient Algorithm

Demystifying SGD with Doubly Stochastic Gradients

Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Optimization

An Asynchronous Gradient Descent Based Method for Distributed Resource Allocation with Bounded Variables

Efficient Algorithms for A Class of Stochastic Hidden Convex Optimization and Its Applications in Network Revenue Management

Delay Reduction Via Lagrange Multipliers in Stochastic Network Optimization

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic

Randomized Block Subgradient Methods for Convex Nonsmooth and Stochastic Optimization

Gradient and Projection Free Distributed Online Min-Max Resource Optimization