Abstract:The last decade witnessed a rise in the importance of supervised learning applications involving {\em big data} and {\em big models}. Big data refers to situations where the amounts of training data available and needed causes difficulties in the training phase of the pipeline. Big model refers to situations where large dimensional and over-parameterized models are needed for the application at hand. Both of these phenomena lead to a dramatic increase in research activity aimed at taming the issues via the design of new sophisticated optimization algorithms. In this paper we turn attention to the {\em big constraints} scenario and argue that elaborate machine learning systems of the future will necessarily need to account for a large number of real-world constraints, which will need to be incorporated in the training process. This line of work is largely unexplored, and provides ample opportunities for future work and applications. To handle the {\em big constraints} regime, we propose a {\em stochastic penalty} formulation which {\em reduces the problem to the well understood big data regime}. Our formulation has many interesting properties which relate it to the original problem in various ways, with mathematical guarantees. We give a number of results specialized to nonconvex loss functions, smooth convex functions, strongly convex functions and convex constraints. We show through experiments that our approach can beat competing approaches by several orders of magnitude when a medium accuracy solution is required.

Portfolio reshaping under 1st order stochastic dominance constraints by the exact penalty function methods

Portfolio reshaping under 1st-order stochastic dominance constraints by the exact penalty function methods

A Novel Penalty Approach for Nonlinear Dynamic Optimization Problems with Inequality Path Constraints

Stochastic Nonlinear Complementarity Problems: Stochastic Programming Reformulation and Penalty-Based Approximation Method

Error bound and exact penalty method for optimization problems with nonnegative orthogonal constraint

Penalty method for the sparse portfolio optimization problem

Penalty Methods for a Class of Non-Lipschitz Optimization Problems

A Stochastic Penalty Model for Convex and Nonconvex Optimization with Big Constraints

On the Gradient Method in One Portfolio Management Problem

A class of smooth exact penalty function methods for optimization problems with orthogonality constraints

First-order algorithms for robust optimization problems via convex-concave saddle-point Lagrangian reformulation

Efficient and Scalable Parametric High-Order Portfolios Design via the Skew-t Distribution

An Exact Penalty Approach for Optimization with Nonnegative Orthogonality Constraints

Testing out-of-sample portfolio performance using second-order stochastic dominance constrained optimization approach

Exact Penalization, Level Function Method, and Modified Cutting-Plane Method for Stochastic Programs with Second Order Stochastic Dominance Constraints

A Relaxed Optimization Approach for Cardinality-Constrained Portfolio Optimization

A penalty decomposition approach for multi-objective cardinality-constrained optimization problems

Multistage stochastic dominance: an application to pension fund management

The Exact Projective Penalty Method for Constrained Optimization

Accelerated First-Order Optimization under Nonlinear Constraints

Central-planned Portfolio Selection, Pareto Frontier, and Pareto Improvement