Abstract:Variational regularization is a classical technique to solve statistical inference tasks and inverse problems, with modern data-driven approaches parameterizing regularizers via deep neural networks showcasing impressive empirical performance. Recent works along these lines learn task-dependent regularizers. This is done by integrating information about the measurements and ground-truth data in an unsupervised, critic-based loss function, where the regularizer attributes low values to likely data and high values to unlikely data. However, there is little theory about the structure of regularizers learned via this process and how it relates to the two data distributions. To make progress on this challenge, we initiate a study of optimizing critic-based loss functions to learn regularizers over a particular family of regularizers: gauges (or Minkowski functionals) of star-shaped bodies. This family contains regularizers that are commonly employed in practice and shares properties with regularizers parameterized by deep neural networks. We specifically investigate critic-based losses derived from variational representations of statistical distances between probability measures. By leveraging tools from star geometry and dual Brunn-Minkowski theory, we illustrate how these losses can be interpreted as dual mixed volumes that depend on the data distribution. This allows us to derive exact expressions for the optimal regularizer in certain cases. Finally, we identify which neural network architectures give rise to such star body gauges and when do such regularizers have favorable properties for optimization. More broadly, this work highlights how the tools of star geometry can aid in understanding the geometry of unsupervised regularizer learning.

The Geometry and Well-Posedness of Sparse Regularized Linear Regression

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Piecewise Linearity of Min-Norm Solution Map of a Nonconvexly Regularized Convex Sparse Model

Sparse bayesian inference with regularized gaussian distributions

Monotonicity in Quadratically Regularized Linear Programs

Characterizing the SLOPE Trade-off: A Variational Perspective and the Donoho-Tanner Limit

High-dimensional Inference Via Lipschitz Sparsity-Yielding Regularizers.

Regularity Properties for Sparse Regression

Does the $\ell_1$-norm Learn a Sparse Graph under Laplacian Constrained Graphical Models?

Analysis of $p$-Laplacian Regularization in Semi-Supervised Learning

The Trimmed Lasso: Sparsity and Robustness

The Star Geometry of Critic-Based Regularizer Learning

On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds

Graph-based regularization for regression problems with alignment and highly-correlated designs

On Regularized Sparse Logistic Regression

Locality Regularized Reconstruction: Structured Sparsity and Delaunay Triangulations

Regularized Matrix Regression

Sparse Recovery Via Nonconvex Regularized M-estimators over ℓ_q-Balls

Nonconvex Sparse Logistic Regression with Weakly Convex Regularization

Structured Regularization for Constrained Optimization on the SPD Manifold

Learning Sparse Visual Representations with Leaky Capped Norm Regularizers