Abstract:Classical penalty methods solve a sequence of unconstrained problems that put greater and greater stress on meeting the constraints. In the limit as the penalty constant tends to $\infty$, one recovers the constrained solution. In the exact penalty method, squared penalties are replaced by absolute value penalties, and the solution is recovered for a finite value of the penalty constant. In practice, the kinks in the penalty and the unknown magnitude of the penalty constant prevent wide application of the exact penalty method in nonlinear programming. In this article, we examine a strategy of path following consistent with the exact penalty method. Instead of performing optimization at a single penalty constant, we trace the solution as a continuous function of the penalty constant. Thus, path following starts at the unconstrained solution and follows the solution path as the penalty constant increases. In the process, the solution path hits, slides along, and exits from the various constraints. For quadratic programming, the solution path is piecewise linear and takes large jumps from constraint to constraint. For a general convex program, the solution path is piecewise smooth, and path following operates by numerically solving an ordinary differential equation segment by segment. Our diverse applications to a) projection onto a convex set, b) nonnegative least squares, c) quadratically constrained quadratic programming, d) geometric programming, and e) semidefinite programming illustrate the mechanics and potential of path following. The final detour to image denoising demonstrates the relevance of path following to regularized estimation in inverse problems. In regularized estimation, one follows the solution path as the penalty constant decreases from a large value.

Lp-Regularized Least Squares (0<p<1) and Critical Path

A Generic Path Algorithm for Regularized Statistical Estimation

A Novel Penalty Approach for Nonlinear Dynamic Optimization Problems with Inequality Path Constraints

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Group Sparse Optimization Via Lp, Q Regularization.

Path Following in the Exact Penalty Method of Convex Programming

An algorithmic view of $\ell_2$ regularization and some path-following algorithms

Characterizing the SLOPE Trade-off: A Variational Perspective and the Donoho-Tanner Limit

Piecewise Linearity of Min-Norm Solution Map of a Nonconvexly Regularized Convex Sparse Model

A Sufficient Condition on Monotonic Increase of the Number of Nonzero Entry in the Optimizer of L1 Norm Penalized Least-Square Problem

A Constrained Optimization Approach for Lcp

L_p-norm Regularization Algorithms for Optimization over Permutation Matrices

Homotopy Based Algorithms for $\ell _{\scriptscriptstyle 0}$ -Regularized Least-Squares

The Well Tempered Lasso

Piecewise Linear Relationship Between L1 Norm Objective Functional Values and L∞ Norm Constraint Specifications

Revisiting $L_q(0\leq q<1)$ Norm Regularized Optimization

Neural Network for a Class of Sparse Optimization with L0-regularization.

On the Properties of the Solution Path of the Constrained and Penalized L2-L0 Problems

Complexity Analysis of Regularization Methods for Implicitly Constrained Least Squares

Partial least trimmed squares regression

Pathwise optimization for bridge-type estimators and its applications