Abstract:Non-smooth optimization is a core ingredient of many imaging or machine learning pipelines. Non-smoothness encodes structural constraints on the solutions, such as sparsity, group sparsity, low-rank and sharp edges. It is also the basis for the definition of robust loss functions and scale-free functionals such as square-root Lasso. Standard approaches to deal with non-smoothness leverage either proximal splitting or coordinate descent. These approaches are effective but usually require parameter tuning, preconditioning or some sort of support pruning. In this work, we advocate and study a different route, which operates a non-convex but smooth over-parametrization of the underlying non-smooth optimization problems. This generalizes quadratic variational forms that are at the heart of the popular Iterative Reweighted Least Squares (IRLS). Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric. This analysis is crucial to derive convergence bounds that are dimension-free. This explains the efficiency of the method when using small grid sizes in imaging. Our main algorithmic contribution is to apply the Variable Projection (VarPro) method which defines a new formulation by explicitly minimizing over part of the variables. This leads to a better conditioning of the minimized functional and improves the convergence of simple but very efficient gradient-based methods, for instance quasi-Newton solvers. We exemplify the use of this new solver for the resolution of regularized regression problems for inverse problems and supervised learning, including total variation prior and non-convex regularizers.

Smoothing Newton Method for $ \ell^0 $-$ \ell^2 $ Regularized Linear Inverse Problem

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Smoothing $\mathcal{L}^2$ gradients in iterative regularization

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Avoiding strict saddle points of nonconvex regularized problems

Semismooth Newton Algorithm for Efficient Projections onto $\ell_1, ∞$-norm Ball

An Efficient Semismooth Newton Method for Adaptive Sparse Signal Recovery Problems

Smooth over-parameterized solvers for non-smooth structured optimization

Semi-Smooth Newton Algorithm for Non-Convex Penalized Linear Regression

A regularized Newton method for $\ell_q$-norm composite optimization problems

Generalized Newton Method with Positive Definite Regularization for Nonsmooth Optimization Problems with Nonisolated Solutions

Semismooth Newton Algorithm for Efficient Projections Onto ℓ1, ∞-Norm Ball

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

A Decoupled Approach for Composite Sparse-plus-Smooth Penalized Optimization

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization

Tikhonov Regularization with L^0-Term Complementing a Convex Penalty: L^1 Convergence under Sparsity Constraints

Adaptive Quadratically Regularized Newton Method for Riemannian Optimization

An iteration regularization method with general convex penalty for nonlinear inverse problems in Banach spaces

Revisiting $L_q(0\leq q<1)$ Norm Regularized Optimization

Inexact FPPA for the $\ell_0$ Sparse Regularization Problem