Abstract:We consider a class of sparsity-inducing optimization problems whose constraint set is regularizer-compatible, in the sense that, the constraint set becomes easy-to-project-onto after a coordinate transformation induced by the sparsity-inducing regularizer. Our model is general enough to cover, as special cases, the ordered LASSO model and its variants with some commonly used nonconvex sparsity-inducing regularizers. The presence of both the sparsity-inducing regularizer and the constraint set poses challenges on the design of efficient algorithms. In this paper, by exploiting absolute-value symmetry and other properties in the sparsity-inducing regularizer, we propose a new algorithm, called the Doubly Majorized Algorithm (DMA), for this class of problems. The DMA makes use of projections onto the constraint set after the coordinate transformation in each iteration, and hence can be performed efficiently. Without invoking any commonly used constraint qualification conditions such as those based on horizon subdifferentials, we show that any accumulation point of the sequence generated by DMA is a so-called $\psi_{\rm opt}$-stationary point, a new notion of stationarity we define as inspired by the notion of $L$-stationarity. We also show that any global minimizer of our model has to be a $\psi_{\rm opt}$-stationary point, again without imposing any constraint qualification conditions. Finally, we illustrate numerically the performance of DMA on solving variants of ordered LASSO with nonconvex regularizers.

Algorithmic Design of Majorizers for Large-Scale Inverse Problems

Universal Majorization-Minimization Algorithms

The appeals of quadratic majorization–minimization

Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning

Composite Optimization by Nonconvex Majorization-Minimization

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

Novel Algorithms based on Majorization Minimization for Nonnegative Matrix Factorization

Nonconvex Optimization via MM Algorithms: Convergence Theory

Relaxed Majorization-Minimization for Non-Smooth and Non-Convex Optimization

Block majorization-minimization with diminishing radius for constrained nonconvex optimization

Distance Majorization and Its Applications

Stochastic Variance-Reduced Majorization-Minimization Algorithms

Generalized Majorization-Minimization for Non-Convex Optimization.

Min-Max Framework for Majorization-Minimization Algorithms in Signal Processing Applications: An Overview

On the Convergence of Block Majorization-Minimization Algorithms on the Grassmann Manifold

Convergence and complexity of block majorization-minimization for constrained block-Riemannian optimization

On the Global Convergence of Majorization Minimization Algorithms for Nonconvex Optimization Problems.

Convergence Analysis of Generalized ADMM with Majorization for Linearly Constrained Composite Convex Optimization

Doubly majorized algorithm for sparsity-inducing optimization problems with regularizer-compatible constraints

Convergence analysis of stochastic higher-order majorization-minimization algorithms