Abstract:The last decade witnessed a rise in the importance of supervised learning applications involving {\em big data} and {\em big models}. Big data refers to situations where the amounts of training data available and needed causes difficulties in the training phase of the pipeline. Big model refers to situations where large dimensional and over-parameterized models are needed for the application at hand. Both of these phenomena lead to a dramatic increase in research activity aimed at taming the issues via the design of new sophisticated optimization algorithms. In this paper we turn attention to the {\em big constraints} scenario and argue that elaborate machine learning systems of the future will necessarily need to account for a large number of real-world constraints, which will need to be incorporated in the training process. This line of work is largely unexplored, and provides ample opportunities for future work and applications. To handle the {\em big constraints} regime, we propose a {\em stochastic penalty} formulation which {\em reduces the problem to the well understood big data regime}. Our formulation has many interesting properties which relate it to the original problem in various ways, with mathematical guarantees. We give a number of results specialized to nonconvex loss functions, smooth convex functions, strongly convex functions and convex constraints. We show through experiments that our approach can beat competing approaches by several orders of magnitude when a medium accuracy solution is required.

A Penalty-Like Neurodynamic Approach to Convex Optimization Problems with Set Constraint

A Novel Penalty Approach for Nonlinear Dynamic Optimization Problems with Inequality Path Constraints

An Effective Simultaneous Approach with Variable Time Nodes for Dynamic Optimization Problems

Neurodynamic approaches for multi-agent distributed optimization

A Neural Network Model for General Minimax Problem

Penalty Methods for a Class of Non-Lipschitz Optimization Problems

A discrete-time neural network for optimization problems with hybrid constraints.

Exact Penalty Function Approach to Constrained Optimal Control Problems

A nonautonomous-differential-inclusion neurodynamic approach for nonsmooth distributed optimization on multi-agent systems

An Improved Multi-Population Genetic Algorithm For Constrained Nonlinear Optimization

A Stochastic Penalty Model for Convex and Nonconvex Optimization with Big Constraints

A Partial Exact Penalty Function Approach for Constrained Optimization

New experimental evidence for mechanism of arrhythmogenic membrane potential alternans based on balance of electrogenic I(NCX)/I(Ca) currents.

An Exact Penalty Approach for Optimization with Nonnegative Orthogonality Constraints

A class of smooth exact penalty function methods for optimization problems with orthogonality constraints

Enhancing neurodynamic approach with physics-informed neural networks for solving non-smooth convex optimization problems

Bayesian Optimization via Exact Penalty

A Recurrent Neural Network Approach for Constrained Distributed Fuzzy Convex Optimization

Huber Loss-Based Penalty Approach to Problems with Linear Constraints

Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints