Abstract:Non-smooth optimization is a core ingredient of many imaging or machine learning pipelines. Non-smoothness encodes structural constraints on the solutions, such as sparsity, group sparsity, low-rank and sharp edges. It is also the basis for the definition of robust loss functions and scale-free functionals such as square-root Lasso. Standard approaches to deal with non-smoothness leverage either proximal splitting or coordinate descent. These approaches are effective but usually require parameter tuning, preconditioning or some sort of support pruning. In this work, we advocate and study a different route, which operates a non-convex but smooth over-parametrization of the underlying non-smooth optimization problems. This generalizes quadratic variational forms that are at the heart of the popular Iterative Reweighted Least Squares (IRLS). Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric. This analysis is crucial to derive convergence bounds that are dimension-free. This explains the efficiency of the method when using small grid sizes in imaging. Our main algorithmic contribution is to apply the Variable Projection (VarPro) method which defines a new formulation by explicitly minimizing over part of the variables. This leads to a better conditioning of the minimized functional and improves the convergence of simple but very efficient gradient-based methods, for instance quasi-Newton solvers. We exemplify the use of this new solver for the resolution of regularized regression problems for inverse problems and supervised learning, including total variation prior and non-convex regularizers.

Enhancing sharp augmented Lagrangian methods with smoothing techniques for nonlinear programming

Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

Implicit augmented Lagrangian and generalized optimization

Nonconvex and Nonsmooth Optimization with Generalized Orthogonality Constraints: an Approximate Augmented Lagrangian Method

An Augmented Lagrangian Primal-Dual Semismooth Newton Method for Multi-Block Composite Optimization

Augmented Lagrangian optimization under fixed-point arithmetic

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

A New Insight on Augmented Lagrangian Method with Applications in Machine Learning

The inexact power augmented Lagrangian method for constrained nonconvex optimization

Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization

An Inexact Augmented Lagrangian Framework for Nonconvex Optimization with Nonlinear Constraints

Smooth over-parameterized solvers for non-smooth structured optimization

An augmented Lagrangian trust-region method with inexact gradient evaluations to accelerate constrained optimization problems using model hyperreduction

A fresh look at nonsmooth Levenberg–Marquardt methods with applications to bilevel optimization

Damped Proximal Augmented Lagrangian Method for weakly-Convex Problems with Convex Constraints

Adaptive Augmented Lagrangian Methods: Algorithms and Practical Numerical Experience

Global Convergence Analysis of the Power Proximal Point and Augmented Lagrangian Method

A Semismooth Newton Based Augmented Lagrangian Method for Nonsmooth Optimization on Matrix Manifolds

A Variable Smoothing for Weakly Convex Composite Minimization with Nonconvex Constraint

A New Penalty Dual-Primal Augmented Lagrangian Method and Its Extensions

On proximal augmented Lagrangian based decomposition methods for dual block-angular convex composite programming problems