Abstract:In this work, we consider solving optimization problems with a stochastic objective and deterministic equality constraints. We propose a Trust-Region Sequential Quadratic Programming method to find both first- and second-order stationary points. Our method utilizes a random model to represent the objective function, which is constructed from stochastic observations of the objective and is designed to satisfy proper adaptive accuracy conditions with a high but fixed probability. To converge to first-order stationary points, our method computes a gradient step in each iteration defined by minimizing a quadratic approximation of the objective subject to a (relaxed) linear approximation of the problem constraints and a trust-region constraint. To converge to second-order stationary points, our method additionally computes an eigen step to explore the negative curvature of the reduced Hessian matrix, as well as a second-order correction step to address the potential Maratos effect, which arises due to the nonlinearity of the problem constraints. Such an effect may impede the method from moving away from saddle points. Both gradient and eigen step computations leverage a novel parameter-free decomposition of the step and the trust-region radius, accounting for the proportions among the feasibility residual, optimality residual, and negative curvature. We establish global almost sure first- and second-order convergence guarantees for our method, and present computational results on CUTEst problems, regression problems, and saddle-point problems to demonstrate its superiority over existing line-search-based stochastic methods.

Global Convergence of a Stochastic Levenberg–Marquardt Algorithm Based on Trust Region

On the Complexity of a Stochastic Levenberg-Marquardt Method

A Stochastic Trust Region Method for Unconstrained Optimization Problems

Trust-Region Sequential Quadratic Programming for Stochastic Optimization with Random Models

Globally Convergent Composite-Step Trust-Region Framework for Model-Based Real-Time Optimization

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

On the Modified Trust Region Algorithm for Nonlinear Equations

A Trust Region Algorithm with Conjugate Gradient Technique for Optimization Problems

A MODIFIED LEVENBERG-MARQUATDT METHOD WITH TRUST-REGION TECHNIQUES

A Levenberg–Marquardt Method for Nonlinear Complementarity Problems Based on Nonmonotone Trust Region and Line Search Techniques

An adaptive simple model trust region algorithm based on new weak secant equations

An Improved Trust Region Algorithm for Nonlinear Equations.

On Convergence Properties of the Modified Trust Region Method under Hölderian Error Bound Condition

Non-monotone Nonlinear Conjugate Gradient Method Based on the Trust Region Technique

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

A Modified Trust Region Algorithm For Nonlinear Equations With New Updating Rule Of Trust Region Radius

Trust Region Methods for Nonconvex Stochastic Optimization Beyond Lipschitz Smoothness

A nonmonotone trust region method for unconstrained optimization

Trust-Region Stochastic Optimization with Variance Reduction Technique

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

Linear convergence of decentralized estimation for statistical estimation using gradient method