Abstract:Non-smooth optimization is a core ingredient of many imaging or machine learning pipelines. Non-smoothness encodes structural constraints on the solutions, such as sparsity, group sparsity, low-rank and sharp edges. It is also the basis for the definition of robust loss functions and scale-free functionals such as square-root Lasso. Standard approaches to deal with non-smoothness leverage either proximal splitting or coordinate descent. These approaches are effective but usually require parameter tuning, preconditioning or some sort of support pruning. In this work, we advocate and study a different route, which operates a non-convex but smooth over-parametrization of the underlying non-smooth optimization problems. This generalizes quadratic variational forms that are at the heart of the popular Iterative Reweighted Least Squares (IRLS). Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric. This analysis is crucial to derive convergence bounds that are dimension-free. This explains the efficiency of the method when using small grid sizes in imaging. Our main algorithmic contribution is to apply the Variable Projection (VarPro) method which defines a new formulation by explicitly minimizing over part of the variables. This leads to a better conditioning of the minimized functional and improves the convergence of simple but very efficient gradient-based methods, for instance quasi-Newton solvers. We exemplify the use of this new solver for the resolution of regularized regression problems for inverse problems and supervised learning, including total variation prior and non-convex regularizers.

One-parametric class of smoothing functions for (WSOCCP) and its Jacobian consistency

An inexact smoothing method for the monotone complementarity problem over symmetric cones

Smoothing penalty approach for solving second-order cone complementarity problems

Improved Smoothing Newton Methods for P0 Nonlinear Complementarity Problems

Superlinear／Quadratic One-step Smoothing Newton Methodf or P0-NCP

Self-concordant Smoothing for Large-Scale Convex Composite Optimization

New smooth weighted complementarity functions and a cubically convergent method for wLCP

A Jacobian smoothing inexact Newton method for solving the nonlinear complementary problem

Fast Bivariate Penalized Splines: the Sandwich Smoother

Superlinear/Quadratic One–step Smoothing Newton Method for P 0–NCP

Computation Algorithm for Convex Semi-infinite Program with Second-Order Cones: Special Analyses for Affine and Quadratic Case

Superlinear/quadratic one-step smoothing Newton method for P 0 -NCP without strict complementarity

A Variable Smoothing for Weakly Convex Composite Minimization with Nonconvex Constraint

Constrained Global Optimization by Smoothing

A Class of Smoothing Modulus-Based Iterative Method for Solving Implicit Complementarity Problems

A Proximal Variable Smoothing for Nonsmooth Minimization Involving Weakly Convex Composite with MIMO Application

Smooth over-parameterized solvers for non-smooth structured optimization

Two classes of third-order weighted compact nonlinear schemes for Hamilton-Jacobi equations

SOLVING NONLINEAR COMPLEMENTARITY PROBLEM BY A SMOOTHING HOMOTOPY METHOD

Second-Order Conic Programming Approach for Wasserstein Distributionally Robust Two-Stage Linear Programs

On the smoothing of the norm objective penalty function for two-cardinality sparse constrained optimization problems