Abstract:Superquantiles have recently gained significant interest as a risk-aware metric for addressing fairness and distribution shifts in statistical learning and decision making problems. This paper introduces a fast, scalable and robust second-order computational framework to solve large-scale optimization problems with superquantile-based constraints. Unlike empirical risk minimization, superquantile-based optimization requires ranking random functions evaluated across all scenarios to compute the tail conditional expectation. While this tail-based feature might seem computationally unfriendly, it provides an advantageous setting for a semismooth-Newton-based augmented Lagrangian method. The superquantile operator effectively reduces the dimensions of the Newton systems since the tail expectation involves considerably fewer scenarios. Notably, the extra cost of obtaining relevant second-order information and performing matrix inversions is often comparable to, and sometimes even less than, the effort required for gradient computation. Our developed solver is particularly effective when the number of scenarios substantially exceeds the number of decision variables. In synthetic problems with linear and convex diagonal quadratic objectives, numerical experiments demonstrate that our method outperforms existing approaches by a large margin: It achieves speeds more than 750 times faster for linear and quadratic objectives than the alternating direction method of multipliers as implemented by OSQP for computing low-accuracy solutions. Additionally, it is up to 25 times faster for linear objectives and 70 times faster for quadratic objectives than the commercial solver Gurobi, and 20 times faster for linear objectives and 30 times faster for quadratic objectives than the Portfolio Safeguard optimization suite for high-accuracy solution computations.

What problem does this paper attempt to address?

The paper aims to address the computational challenges associated with large-scale optimization problems with superquantile constraints. Specifically, the paper proposes a fast, scalable, and robust second-order computational framework for solving large-scale convex optimization problems with superquantile constraints. Unlike traditional empirical risk minimization methods, superquantile optimization requires sorting random functions across all scenarios to compute the tail conditional expectation, which is computationally complex. However, this tail-feature-based approach provides a favorable setting for the semismooth-Newton-based augmented Lagrangian method, as the superquantile operator can effectively reduce the dimension of the Newton system. The main contributions of the paper include: 1. **Efficient computational framework**: A new solver is developed that can efficiently solve optimization problems with superquantile constraints at both low and high precision levels. 2. **Significant performance improvement**: Numerical experiments demonstrate that this method is hundreds to thousands of times faster than existing methods such as OSQP and Gurobi when dealing with linear and convex diagonal quadratic objective functions. 3. **Implicit scenario reduction**: By utilizing second-order variational analysis to automatically adjust key scenarios, implicit scenario reduction is achieved, further enhancing computational efficiency. 4. **Wide applicability**: The method performs exceptionally well in handling problems with multiple superquantile constraints or high risk aversion, making it suitable for risk-aware optimization problems in various fields such as engineering design, statistics, and machine learning. In summary, the paper addresses the computational bottleneck in superquantile-constrained optimization problems by proposing an efficient computational framework, providing strong support for practical applications.

Fast Computation of Superquantile-Constrained Optimization Through Implicit Scenario Reduction

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

Efficient Scenario Generation for Heavy-tailed Chance Constrained Optimization

Optimized Scenario Reduction: Solving Large-Scale Stochastic Programs with Quality Guarantees

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates

An Empirical Quantile Estimation Approach to Nonlinear Optimization Problems with Chance Constraints

An Empirical Quantile Estimation Approach for Chance-Constrained Nonlinear Optimization Problems

No-Regret Constrained Bayesian Optimization of Noisy and Expensive Hybrid Models using Differentiable Quantile Function Approximations

An Efficient Penalty Function Method for Scenario-Based Uncertainty Quantification Problems

A Stochastic Approximation Method for Simulation-Based Quantile Optimization

Sequential Quadratic Optimization for Stochastic Optimization with Deterministic Nonlinear Inequality and Equality Constraints

A Multi-Level Simulation Optimization Approach for Quantile Functions

Non-convex scenario optimization

A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobians

A Stochastic Inexact Sequential Quadratic Optimization Algorithm for Nonlinear Equality-Constrained Optimization

Partial-Quasi-Newton Methods: Efficient Algorithms for Minimax Optimization Problems with Unbalanced Dimensionality

Efficient Algorithms for Distributionally Robust Stochastic Optimization with Discrete Scenario Support

A superlinearly convergent SQP algorithm for constrained optimization problems

Functional Constrained Optimization for Risk Aversion and Sparsity Control

On the Performance of SQP Methods for Nonlinear Optimization

A sequential quadratic programming algorithm for equality-constrained optimization without derivatives