Abstract:Non-smooth optimization is a core ingredient of many imaging or machine learning pipelines. Non-smoothness encodes structural constraints on the solutions, such as sparsity, group sparsity, low-rank and sharp edges. It is also the basis for the definition of robust loss functions and scale-free functionals such as square-root Lasso. Standard approaches to deal with non-smoothness leverage either proximal splitting or coordinate descent. These approaches are effective but usually require parameter tuning, preconditioning or some sort of support pruning. In this work, we advocate and study a different route, which operates a non-convex but smooth over-parametrization of the underlying non-smooth optimization problems. This generalizes quadratic variational forms that are at the heart of the popular Iterative Reweighted Least Squares (IRLS). Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric. This analysis is crucial to derive convergence bounds that are dimension-free. This explains the efficiency of the method when using small grid sizes in imaging. Our main algorithmic contribution is to apply the Variable Projection (VarPro) method which defines a new formulation by explicitly minimizing over part of the variables. This leads to a better conditioning of the minimized functional and improves the convergence of simple but very efficient gradient-based methods, for instance quasi-Newton solvers. We exemplify the use of this new solver for the resolution of regularized regression problems for inverse problems and supervised learning, including total variation prior and non-convex regularizers.

A variable smoothing for Nonconvexly constrained nonsmooth optimization with application to sparse spectral clustering

A Variable Smoothing for Weakly Convex Composite Minimization with Nonconvex Constraint

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

A Proximal Variable Smoothing for Nonsmooth Minimization Involving Weakly Convex Composite with MIMO Application

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

Smooth over-parameterized solvers for non-smooth structured optimization

Smoothing Neural Network for Constrained Non-Lipschitz Optimization with Applications

Constrained Global Optimization by Smoothing

An Alternating Manifold Proximal Gradient Method for Sparse Principal Component Analysis and Sparse Canonical Correlation Analysis

A feasible smoothing accelerated projected gradient method for nonsmooth convex optimization

A Smoothing Stochastic Gradient Method for Composite Optimization

A Smoothing Direct Search Method for Monte Carlo-Based Bound Constrained Composite Nonsmooth Optimization.

Self-concordant Smoothing for Large-Scale Convex Composite Optimization

A Smoothing Proximal Gradient Algorithm for Nonsmooth Convex Regression with Cardinality Penalty.

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

Neural Network for Nonsmooth, Nonconvex Constrained Minimization Via Smooth Approximation.

Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization

Stochastic Optimization for Non-convex Inf-Projection Problems

Spectrally Constrained Optimization

Dynamic Anisotropic Smoothing for Noisy Derivative-Free Optimization

A Stochastic Proximal Alternating Minimization for Nonsmooth and Nonconvex Optimization