On the Properties of the Solution Path of the Constrained and Penalized L2-L0 Problems

Junbo Duan,Charles Soussen,David Brie,Jérôme Idier

2009-01-01

Abstract:Technical report on the properties of the L0-constrained least-square minimization problem and the L0-penalized least-square minimization problem: domain of optimization, notion of solution path, properties of the penalized solution path...

What problem does this paper attempt to address?

A Novel Penalty Approach for Nonlinear Dynamic Optimization Problems with Inequality Path Constraints

Xinggao Liu,Yunqing Hu,Jianghua Feng,Kean Liu

DOI: https://doi.org/10.1109/tac.2014.2317293

IF: 6.549

2014-01-01

IEEE Transactions on Automatic Control

Abstract:The control vector parameterization (CVP) approach is popular to solve nonlinear dynamic optimization problems, but handling inequality path constraints within its framework is a challenge. This study deals with inequality path constraints by using the l(1) penalty function and a novel smoothing technique. After clarifying some theories of the proposed approach, a concomitant algorithm is also put forward and verified.
Penalty Decomposition Methods for $L0$-Norm Minimization

Zhaosong Lu,Yong Zhang

DOI: https://doi.org/10.48550/arXiv.1008.5372

2012-05-12

Abstract:In this paper we consider general l0-norm minimization problems, that is, the problems with l0-norm appearing in either objective function or constraint. In particular, we first reformulate the l0-norm constrained problem as an equivalent rank minimization problem and then apply the penalty decomposition (PD) method proposed in [33] to solve the latter problem. By utilizing the special structures, we then transform all matrix operations of this method to vector operations and obtain a PD method that only involves vector operations. Under some suitable assumptions, we establish that any accumulation point of the sequence generated by the PD method satisfies a first-order optimality condition that is generally stronger than one natural optimality condition. We further extend the PD method to solve the problem with the l0-norm appearing in objective function. Finally, we test the performance of our PD methods by applying them to compressed sensing, sparse logistic regression and sparse inverse covariance selection. The computational results demonstrate that our methods generally outperform the existing methods in terms of solution quality and/or speed.

Optimization and Control,Computer Vision and Pattern Recognition,Information Theory,Machine Learning,Numerical Analysis,Methodology
A Sufficient Condition on Monotonic Increase of the Number of Nonzero Entry in the Optimizer of L1 Norm Penalized Least-Square Problem

J. Duan,Charles Soussen,David Brie,Jerome Idier,Y. -P. Wang

DOI: https://doi.org/10.48550/arxiv.1104.3792

2011-01-01

Abstract:The ℓ-1 norm based optimization is widely used in signal processing, especially in recent compressed sensing theory. This paper studies the solution path of the ℓ-1 norm penalized least-square problem, whose constrained form is known as Least Absolute Shrinkage and Selection Operator (LASSO). A solution path is the set of all the optimizers with respect to the evolution of the hyperparameter (Lagrange multiplier). The study of the solution path is of great significance in viewing and understanding the profile of the tradeoff between the approximation and regularization terms. If the solution path of a given problem is known, it can help us to find the optimal hyperparameter under a given criterion such as the Akaike Information Criterion. In this paper we present a sufficient condition on ℓ-1 norm penalized least-square problem. Under this sufficient condition, the number of nonzero entries in the optimizer or solution vector increases monotonically when the hyperparameter decreases. We also generalize the result to the often used total variation case, where the ℓ-1 norm is taken over the first order derivative of the solution vector. We prove that the proposed condition has intrinsic connections with the condition given by Donoho, et al and the positive cone condition by Efron el al . However, the proposed condition does not need to assume the sparsity level of the signal as required by Donoho et al's condition, and is easier to verify than Efron, et al's positive cone condition when being used for practical applications.
Solving the L1 regularized least square problem via a box-constrained smooth minimization

Majid Mohammadi,Wout Hofman,Yaohua Tan,S. Hamid Mousavi

DOI: https://doi.org/10.48550/arXiv.1704.03443

2021-10-20

Abstract:In this paper, an equivalent smooth minimization for the L1 regularized least square problem is proposed. The proposed problem is a convex box-constrained smooth minimization which allows applying fast optimization methods to find its solution. Further, it is investigated that the property "the dual of dual is primal" holds for the L1 regularized least square problem. A solver for the smooth problem is proposed, and its affinity to the proximal gradient is shown. Finally, the experiments on L1 and total variation regularized problems are performed, and the corresponding results are reported.

Optimization and Control,Computer Vision and Pattern Recognition
Smoothing Neural Network for L 0 Regularized Optimization Problem with General Convex Constraints

Wenjing Li,Wei Bian

DOI: https://doi.org/10.1016/j.neunet.2021.08.001

IF: 7.8

2021-01-01

Neural Networks

Abstract:In this paper, we propose a neural network modeled by a differential inclusion to solve a class of discontinuous and nonconvex sparse regression problems with general convex constraints, whose objective function is the sum of a convex but not necessarily differentiable loss function and L 0 regularization. We construct a smoothing relaxation function of L 0 regularization and propose a neural network to solve the considered problem. We prove that the solution of proposed neural network with any initial point satisfying linear equality constraints is global existent, bounded and reaches the feasible region in finite time and remains there thereafter. Moreover, the solution of proposed neural network is its slow solution and any accumulation point of it is a Clarke stationary point of the brought forward nonconvex smoothing approximation problem. In the box-constrained case, all accumulation points of the solution own a unified lower bound property and have a common support set. Except for a special case, any accumulation point of the solution is a local minimizer of the considered problem. In particular, the proposed neural network has a simple structure than most existing neural networks for solving the locally Lipschitz continuous but nonsmooth nonconvex problems. Finally, we give some numerical experiments to show the efficiency of proposed neural network.
Penalty Methods for a Class of Non-Lipschitz Optimization Problems

Xiaojun Chen,Zhaosong Lu,Ting Kei Pong

DOI: https://doi.org/10.1137/15m1028054

IF: 2.763

2016-01-01

SIAM Journal on Optimization

Abstract:We consider a class of constrained optimization problems with a possibly nonconvex non-Lipschitz objective and a convex feasible set being the intersection of a polyhedron and a possibly degenerate ellipsoid. Such problems have a wide range of applications in data science, where the objective is used for inducing sparsity in the solutions while the constraint set models the noise tolerance and incorporates other prior information for data fitting. To solve this class of constrained optimization problems, a common approach is the penalty method. However, there is little theory on exact penalization for problems with nonconvex and non-Lipschitz objective functions. In this paper, we study the existence of exact penalty parameters regarding local minimizers, stationary points, and $\epsilon$-minimizers under suitable assumptions. Moreover, we discuss a penalty method whose subproblems are solved via a nonmonotone proximal gradient method with a suitable update scheme for the penalty parameters and prove the convergence of the algorithm to a KKT point of the constrained problem. Preliminary numerical results demonstrate the efficiency of the penalty method for finding sparse solutions of underdetermined linear systems.

mathematics, applied
ℓ 2 -ℓ 0 Regularization Path Tracking Algorithms

Charles Soussen,Jérôme Idier,Junbo Duan,David Brie

2014-01-01

Abstract:Sparse signal approximation can be formulated as the mixed L2-L0 minimization problem min_x J(x;lambda)=||y-Ax||_2^2+lambda||x||_0. We propose two heuristic search algorithms to minimize J for a continuum of lambda-values, yielding a sequence of coarse to fine approximations. Continuation Single Best Replacement is a bidirectional greedy algorithm adapted from the Single Best Replacement algorithm previously proposed for minimizing J for fixed lambda. L0 regularization path track is a more complex algorithm exploiting that the L2-L0 regularization path is piecewise constant with respect to lambda. Tracking the L0 regularization path is done in a sub-optimal manner by maintaining (i) a list of subsets that are candidates to be solution supports for decreasing lambda's and (ii) the list of critical lambda-values around which the solution changes. Both algorithms gradually construct the L0 regularization path by performing single replacements, i.e., adding or removing a dictionary atom from a subset. A straightforward adaptation of these algorithms yields sub-optimal solutions to min_x ||y-Ax||_2^2 subject to ||x||_0 =0 and to min_x ||x||_0 subject to ||y-Ax||_2^2<= epsilon for continuous values of epsilon. Numerical simulations show the effectiveness of the algorithms on a difficult sparse deconvolution problem inducing a highly correlated dictionary A.
Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Hu Yaohua,Li Chong,Meng Kaiwen,Yang Xiaoqi

DOI: https://doi.org/10.1007/s10898-020-00955-3

2020-01-01

Journal of Global Optimization

Abstract:The ℓ _p regularization problem with 0< p< 1 has been widely studied for finding sparse solutions of linear inverse problems and gained successful applications in various mathematics and applied science fields. The proximal gradient algorithm is one of the most popular algorithms for solving the ℓ _p regularisation problem. In the present paper, we investigate the linear convergence issue of one inexact descent method and two inexact proximal gradient algorithms (PGA). For this purpose, an optimality condition theorem is explored to provide the equivalences among a local minimum, second-order optimality condition and second-order growth property of the ℓ _p regularization problem. By virtue of the second-order optimality condition and second-order growth property, we establish the linear convergence properties of the inexact descent method and inexact PGAs under some simple assumptions. Both linear convergence to a local minimal value and linear convergence to a local minimum are provided. Finally, the linear convergence results of these methods are extended to the infinite-dimensional Hilbert spaces. Our results cannot be established under the framework of Kurdyka–Łojasiewicz theory.
On NP-Hardness of $L_1/L_2$ Minimization and Bound Theory of Nonzero Entries in Solutions

Min Tao,Xiao-Ping Zhang,Yun-Bin Zhao

2024-10-30

Abstract:The $L_1/L_2$ norm ratio has gained significant attention as a measure of sparsity due to three merits: sharper approximation to the $L_0$ norm compared to the $L_1$ norm, being parameter-free and scale-invariant, and exceptional performance with highly coherent matrices. These properties have led to its successful application across a wide range of fields. While several efficient algorithms have been proposed to compute stationary points for $L_1/L_2$ minimization problems, their computational complexity has remained open. In this paper, we prove that finding the global minimum of both constrained and unconstrained $L_1/L_2$ models is strongly NP-hard. In addition, we establish uniform upper bounds on the $L_2$ norm for any local minimizer of both constrained and unconstrained $L_1/L_2$ minimization models. We also derive upper and lower bounds on the magnitudes of the nonzero entries in any local minimizer of the unconstrained model, aiding in classifying nonzero entries. Finally, we extend our analysis to demonstrate that the constrained and unconstrained $L_p/L_q$ ($0 < p \leq 1, 1 < q < +\infty$) models are also strongly NP-hard.

Optimization and Control
Lp-Regularized Least Squares (0<p<1) and Critical Path

Masahiro Yukawa,Shun-ichi Amari

DOI: https://doi.org/10.48550/arXiv.1304.6591

2013-04-24

Information Theory

Abstract:The least squares problem is formulated in terms of Lp quasi-norm regularization (0<p<1). Two formulations are considered: (i) an Lp-constrained optimization and (ii) an Lp-penalized (unconstrained) optimization. Due to the nonconvexity of the Lp quasi-norm, the solution paths of the regularized least squares problem are not ensured to be continuous. A critical path, which is a maximal continuous curve consisting of critical points, is therefore considered separately. The critical paths are piecewise smooth, as can be seen from the viewpoint of the variational method, and generally contain non-optimal points such as saddle points and local maxima as well as global/local minima. Along each critical path, the correspondence between the regularization parameters (which govern the 'strength' of regularization in the two formulations) is non-monotonic and, more specifically, it has multiplicity. Two paths of critical points connecting the origin and an ordinary least squares (OLS) solution are highlighted. One is a main path starting at an OLS solution, and the other is a greedy path starting at the origin. Part of the greedy path can be constructed with a generalized Minkowskian gradient. The breakpoints of the greedy path coincide with the step-by-step solutions generated by using orthogonal matching pursuit (OMP), thereby establishing a direct link between OMP and Lp-regularized least squares.
A Constrained Optimization Approach for Lcp

JL Zhang,J Chen,XJ Zhuo

2004-01-01

Journal of Computational Mathematics

Abstract:In this paper, LCP is converted to an equivalent nonsmooth nonlinear equation system H(x, y) = 0 by using the famous NCP function-Fischer-Burmeister function. Note that some equations in H(x, y) = 0 are nonsmooth and nonlinear hence difficult to solve while the others are linear hence easy to solve. Then we further convert the nonlinear equation system H (x, y) = 0 to an optimization problem with linear equality constraints. After that we study the conditions under which the K-T points of the optimization problem are the solutions of the original LCP and propose a method to solve the optimization problem. In this algorithm, the search direction is obtained by solving a strict convex programming at each iterative point. However, our algorithm is essentially different from traditional SQP method. The global convergence of the method is proved under mild conditions. In addition, we can prove that the algorithm is convergent superlinearly under the conditions: M is P-0 matrix and the limit point is a strict complementarity solution of LCP. Preliminary numerical experiments are reported with this method.
Efficient Projected Gradient Methods for L 0 Constrained Optimization

Fengmin Xu,Yu‐Hong Dai,Zhihua Zhao,Zongben Xu

2021-01-01

Abstract:Sparse optimization has attracted increasing attention in numerous areas such as compressed sensing, financial optimization and image processing. In this paper, we first consider a special class of l0 constrained optimization problems, which involves box constraints and a singly linear constraint. An efficient approach is provided for calculating the projection over the feasibility set after a careful analysis on the projection subproblem. Then we present several types of projected gradient methods for a general class of l0 constrained optimization problems. Global convergence of the methods are established under suitable assumptions. Finally, we illustrate some applications of the proposed methods for signal recovery and index tracking. Especially for index tracking, we propose a new model subject to an adaptive upper bound on the sparse portfolio weights. The computational results demonstrate that the proposed projected gradient methods are efficient in terms of solution quality.
Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

Wenqiang Pu,Kaizhao Sun,Jiawei Zhang

2024-08-27

Abstract:This paper introduces a smoothed proximal Lagrangian method for minimizing a nonconvex smooth function over a convex domain with additional explicit convex nonlinear constraints. Two key features are 1) the proposed method is single-looped, and 2) an first-order iteration complexity of $\mathcal{O}(\epsilon^{-2})$ is established under mild regularity assumptions. The first feature suggests the practical efficiency of the proposed method, while the second feature highlights its theoretical superiority. Numerical experiments on various problem scales demonstrate the advantages of the proposed method in terms of speed and solution quality.

Optimization and Control
Lower Bound Theory of Nonzero Entries in Solutions of $\Ell_2$-$\Ell_p$ Minimization

Xiaojun Chen,Fengmin Xu,Yinyu Ye

DOI: https://doi.org/10.1137/090761471

IF: 2.968

2010-01-01

SIAM Journal on Scientific Computing

Abstract:Recently, variable selection and sparse reconstruction are solved by finding an optimal solution of a minimization model, where the objective function is the sum of a data-fitting term in l(2) norm and a regularization term in l(p) norm (0 < p < 1). Since it is a nonconvex model, most algorithms for solving the problem can provide only an approximate local optimal solution, where nonzero entries in the solution cannot be identified theoretically. In this paper, we establish lower bounds for the absolute value of nonzero entries in every local optimal solution of the model, which can be used to indentify zero entries precisely in any numerical solution. Therefore, we have developed a lower bound theorem to classify zero and nonzero entries in every local solution. These lower boundsclearly show the relationship between the sparsity of the solution and the choice of the regularization parameter and norm so that our theorem can be used for selecting desired model parameters and norms. Furthermore, we also develop error bounds for verifying the accuracy of numerical solutions of the l(2)-(p) minimization model. To demonstrate applications of our theory, we propose a hybrid orthogonal matching pursuit-smoothing gradient (OMP-SG) method for solving the nonconvex, non-Lipschitz continuous l(2)-l(p) minimization problem. Computational results show the effectiveness of the lower bounds for identifying nonzero entries in numerical solutions and the OMP-SG method for finding a high quality numerical solution.
Optimality and Complexity for Constrained Optimization Problems with Nonconvex Regularization.

Wei Bian,Xiaojun Chen

DOI: https://doi.org/10.1287/moor.2016.0837

IF: 2.215

2017-01-01

Mathematics of Operations Research

Abstract:In this paper, we consider a class of constrained optimization problems where the feasible set is a general closed convex set, and the objective function has a nonsmooth, nonconvex regularizer. Such a regularizer includes widely used SCAD, MCP, logistic, fraction, hard thresholding, and non-Lipschitz L p penalties as special cases. Using the theory of the generalized directional derivative and the tangent cone, we derive a first order necessary optimality condition for local minimizers of the problem, and define the generalized stationary point of it. We show that the generalized stationary point is the Clarke stationary point when the objective function is Lipschitz continuous at this point, and satisfies the existing necessary optimality conditions when the objective function is not Lipschitz continuous at this point. Moreover, we prove the consistency between the generalized directional derivative and the limit of the classic directional derivatives associated with the smoothing function. Finally, we establish a lower bound property for every local minimizer and show that finding a global minimizer is strongly NP-hard when the objective function has a concave regularizer.
A Path-Based Approach to Constrained Sparse Optimization

Nadav Hallak

DOI: https://doi.org/10.1137/22m1535498

IF: 2.763

2024-02-22

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 1, Page 790-816, March 2024. This paper proposes a path-based approach for the minimization of a continuously differentiable function over sparse symmetric sets, which is a hard problem that exhibits a restrictiveness-hierarchy of necessary optimality conditions. To achieve the more restrictive conditions in the hierarchy, state-of-the-art algorithms require a support optimization oracle that must exactly solve the problem in smaller dimensions. The path-based approach developed in this study produces a path-based optimality condition, which is placed well in the restrictiveness-hierarchy, and a method to achieve it that does not require a support optimization oracle and, moreover, is projection-free. In the development process, new results are derived for the regularized linear minimization problem over sparse symmetric sets, which give additional means to identify optimal solutions for convex and concave objective functions. We complement our results with numerical examples.

mathematics, applied
Constrained optimization in classes of analytic functions with prescribed pointwise values

Laurent Baratchart,Juliette Leblond,Dmitry Ponomarev

DOI: https://doi.org/10.48550/arXiv.1401.7633

2014-01-29

Analysis of PDEs

Abstract:We consider an overdetermined problem for Laplace equation on a disk with partial boundary data where additional pointwise data inside the disk have to be taken into account. After reformulation, this ill-posed problem reduces to a bounded extremal problem of best norm-constrained approximation of partial L2 boundary data by traces of holomorphic functions which satisfy given pointwise interpolation conditions. The problem of best norm-constrained approximation of a given L2 function on a subset of the circle by the trace of a H2 function has been considered in [Baratchart \& Leblond, 1998]. In the present work, we extend such a formulation to the case where the additional interpolation conditions are imposed. We also obtain some new results that can be applied to the original problem: we carry out stability analysis and propose a novel method of evaluation of the approximation and blow-up rates of the solution in terms of a Lagrange parameter leading to a highly-efficient computational algorithm for solving the problem.
Error bound and exact penalty method for optimization problems with nonnegative orthogonal constraint

Yitian Qian,Shaohua Pan,Lianghai Xiao

DOI: https://doi.org/10.1093/imanum/drac084

2023-02-02

IMA Journal of Numerical Analysis

Abstract:Abstract This paper is concerned with a class of optimization problems with the non-negative orthogonal constraint, in which the objective function is $L$-smooth on an open set containing the Stiefel manifold $\textrm {St}(n,r)$. We derive a locally Lipschitzian error bound for the feasible points without zero rows when $n>r>1$, and when $n>r=1$ or $n=r$ achieve a global Lipschitzian error bound. Then, we show that the penalty problem induced by the elementwise $\ell _1$-norm distance to the non-negative cone is a global exact penalty, and so is the one induced by its Moreau envelope under a lower second-order calmness of the objective function. A practical penalty algorithm is developed by solving approximately a series of smooth penalty problems with a retraction-based nonmonotone line-search proximal gradient method, and any cluster point of the generated sequence is shown to be a stationary point of the original problem. Numerical comparisons with the ALM [Wen, Z. W. & Yin, W. T. (2013, A feasible method for optimization with orthogonality constraints. Math. Programming, 142, 397–434),] and the exact penalty method [Jiang, B., Meng, X., Wen, Z. W. & Chen, X. J. (2022, An exact penalty approach for optimization with nonnegative orthogonality constraints. Math. Programming. https://doi.org/10.1007/s10107-022-01794-8)] indicate that our penalty method has an advantage in terms of the quality of solutions despite taking a little more time.

mathematics, applied
Complexity Analysis of Regularization Methods for Implicitly Constrained Least Squares

Akwum Onwunta,Clément W. Royer

DOI: https://doi.org/10.1007/s10915-024-02691-2

2024-10-20

Journal of Scientific Computing

Abstract:Optimization problems constrained by partial differential equations (PDEs) naturally arise in scientific computing, as those constraints often model physical systems or the simulation thereof. In an implicitly constrained approach, the constraints are incorporated into the objective through a reduced formulation. To this end, a numerical procedure is typically applied to solve the constraint system, and efficient numerical routines with quantifiable cost have long been developed for that purpose. Meanwhile, the field of complexity in optimization, that estimates the cost of an optimization algorithm, has received significant attention in the literature, with most of the focus being on unconstrained or explicitly constrained problems. In this paper, we analyze an algorithmic framework based on quadratic regularization for implicitly constrained nonlinear least squares. By leveraging adjoint formulations, we can quantify the worst-case cost of our method to reach an approximate stationary point of the optimization problem. Our definition of such points exploits the least-squares structure of the objective, and provides new complexity insights even in the unconstrained setting. Numerical experiments conducted on PDE-constrained optimization problems demonstrate the efficiency of the proposed framework.

mathematics, applied

On the Properties of the Solution Path of the Constrained and Penalized L2-L0 Problems

A Novel Penalty Approach for Nonlinear Dynamic Optimization Problems with Inequality Path Constraints

Penalty Decomposition Methods for $L0$-Norm Minimization

A Sufficient Condition on Monotonic Increase of the Number of Nonzero Entry in the Optimizer of L1 Norm Penalized Least-Square Problem

Solving the L1 regularized least square problem via a box-constrained smooth minimization

Smoothing Neural Network for L 0 Regularized Optimization Problem with General Convex Constraints

Penalty Methods for a Class of Non-Lipschitz Optimization Problems

ℓ 2 -ℓ 0 Regularization Path Tracking Algorithms

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

On NP-Hardness of $L_1/L_2$ Minimization and Bound Theory of Nonzero Entries in Solutions

Lp-Regularized Least Squares (0<p<1) and Critical Path

A Constrained Optimization Approach for Lcp

Efficient Projected Gradient Methods for L 0 Constrained Optimization

Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

Lower Bound Theory of Nonzero Entries in Solutions of $\Ell_2$-$\Ell_p$ Minimization

Optimality and Complexity for Constrained Optimization Problems with Nonconvex Regularization.

A Path-Based Approach to Constrained Sparse Optimization

Constrained optimization in classes of analytic functions with prescribed pointwise values

Error bound and exact penalty method for optimization problems with nonnegative orthogonal constraint

Complexity Analysis of Regularization Methods for Implicitly Constrained Least Squares