Abstract:Non-smooth optimization is a core ingredient of many imaging or machine learning pipelines. Non-smoothness encodes structural constraints on the solutions, such as sparsity, group sparsity, low-rank and sharp edges. It is also the basis for the definition of robust loss functions and scale-free functionals such as square-root Lasso. Standard approaches to deal with non-smoothness leverage either proximal splitting or coordinate descent. These approaches are effective but usually require parameter tuning, preconditioning or some sort of support pruning. In this work, we advocate and study a different route, which operates a non-convex but smooth over-parametrization of the underlying non-smooth optimization problems. This generalizes quadratic variational forms that are at the heart of the popular Iterative Reweighted Least Squares (IRLS). Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric. This analysis is crucial to derive convergence bounds that are dimension-free. This explains the efficiency of the method when using small grid sizes in imaging. Our main algorithmic contribution is to apply the Variable Projection (VarPro) method which defines a new formulation by explicitly minimizing over part of the variables. This leads to a better conditioning of the minimized functional and improves the convergence of simple but very efficient gradient-based methods, for instance quasi-Newton solvers. We exemplify the use of this new solver for the resolution of regularized regression problems for inverse problems and supervised learning, including total variation prior and non-convex regularizers.

Interplay between preconditioning and regularization for linear ill-posed problems solved by conjugate gradient. Application to optical flow estimation

Multigrid preconditioning for regularized least-squares problems

Polynomial Preconditioners for Regularized Linear Inverse Problems

A preconditioned Krylov subspace method for linear inverse problems with general-form Tikhonov regularization

A fast iterative regularization method for ill-posed problems

A Preconditioned Version of a Nested Primal-Dual Algorithm for Image Deblurring

A preconditioning for the spectral solution of incompressible variable-density flows

Smooth over-parameterized solvers for non-smooth structured optimization

Nonlocal sparse and low-rank regularization for optical flow estimation.

Regularization of linear ill-posed problems involving multiplication operators

Generalized Structure Preserving Preconditioners for Frame-Based Image Deblurring

Smoothing $\mathcal{L}^2$ gradients in iterative regularization

Regularization of linear inverse problems with irregular noise using embedding operators

Preconditioning for PDE-constrained optimization with total variation regularization

Regularization properties of dual subgradient flow

Influence of the order between discretization and regularization in solving ill-posed problems

A rational conjugate gradient method for linear ill-conditioned problems

A robust and time-parallel preconditioner for parabolic reconstruction problems using Isogeometric Analysis

Modified iterated Tikhonov methods for solving systems of nonlinear ill-posed equations

Learning a generalized multiscale prolongation operator

A New Highly Anti-Interference Regularization Method for Ill-Posed Problems