Abstract:Non-smooth optimization is a core ingredient of many imaging or machine learning pipelines. Non-smoothness encodes structural constraints on the solutions, such as sparsity, group sparsity, low-rank and sharp edges. It is also the basis for the definition of robust loss functions and scale-free functionals such as square-root Lasso. Standard approaches to deal with non-smoothness leverage either proximal splitting or coordinate descent. These approaches are effective but usually require parameter tuning, preconditioning or some sort of support pruning. In this work, we advocate and study a different route, which operates a non-convex but smooth over-parametrization of the underlying non-smooth optimization problems. This generalizes quadratic variational forms that are at the heart of the popular Iterative Reweighted Least Squares (IRLS). Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric. This analysis is crucial to derive convergence bounds that are dimension-free. This explains the efficiency of the method when using small grid sizes in imaging. Our main algorithmic contribution is to apply the Variable Projection (VarPro) method which defines a new formulation by explicitly minimizing over part of the variables. This leads to a better conditioning of the minimized functional and improves the convergence of simple but very efficient gradient-based methods, for instance quasi-Newton solvers. We exemplify the use of this new solver for the resolution of regularized regression problems for inverse problems and supervised learning, including total variation prior and non-convex regularizers.

Smoothing Proximal Gradient Method for General Structured Sparse Learning

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

A Smoothing Proximal Gradient Algorithm for Nonsmooth Convex Regression with Cardinality Penalty.

Efficient Inexact Proximal Gradient Algorithms for Structured Sparsity-Inducing Norm.

An Alternating Proximal Splitting Method With Global Convergence For Nonconvex Structured Sparsity Optimization

Faster Training Algorithms for Structured Sparsity-Inducing Norm

Smoothing composite proximal gradient algorithm for sparse group Lasso problems with nonsmooth loss functions

Proximal Iteration for Nonlinear Adaptive Lasso

Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization

A Modified Proximal Gradient Method for a Class of Sparse Optimization Problems

Nonconvex Sparse Logistic Regression Via Proximal Gradient Descent.

A Proximal Gradient Method for Regularized Deep Neural Networks

Learning Model-Based Sparsity via Projected Gradient Descent

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

Smooth over-parameterized solvers for non-smooth structured optimization

A Decoupled Approach for Composite Sparse-plus-Smooth Penalized Optimization

A first-order optimization algorithm for statistical learning with hierarchical sparsity structure

A General Iterative Shrinkage and Thresholding Algorithm for Non-convex Regularized Optimization Problems

Generalized non-convex non-smooth sparse and low rank minimization using proximal average.

Accelerated gradient methods for sparse statistical learning with nonconvex penalties