Abstract:Recently, there has been a surge in interest in developing optimization algorithms for overparameterized models as achieving generalization is believed to require algorithms with suitable biases. This interest centers on minimizing sharpness of the original loss function; the Sharpness-Aware Minimization (SAM) algorithm has proven effective. However, most literature only considers a few sharpness measures, such as the maximum eigenvalue or trace of the training loss Hessian, which may not yield meaningful insights for non-convex optimization scenarios like neural networks. Additionally, many sharpness measures are sensitive to parameter invariances in neural networks, magnifying significantly under rescaling parameters. Motivated by these challenges, we introduce a new class of sharpness measures in this paper, leading to new sharpness-aware objective functions. We prove that these measures are \textit{universally expressive}, allowing any function of the training loss Hessian matrix to be represented by appropriate hyperparameters. Furthermore, we show that the proposed objective functions explicitly bias towards minimizing their corresponding sharpness measures, and how they allow meaningful applications to models with parameter invariances (such as scale-invariances). Finally, as instances of our proposed general framework, we present \textit{Frob-SAM} and \textit{Det-SAM}, which are specifically designed to minimize the Frobenius norm and the determinant of the Hessian of the training loss, respectively. We also demonstrate the advantages of our general framework through extensive experiments.

Uniform Weak Sharp Minima for Multiobjective Optimization Problems

Weak Sharp Minima on Riemannian Manifolds

Weak Sharp Minima for Convex Infinite Optimization Problems in Normed Linear Spaces

The augmented weak sharpness of solution sets in equilibrium problems

Weak sharp minima at infinity and solution stability in mathematical programming via asymptotic analysis

On the weakest constraint qualification for sharp local minimizers

Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learning

The Multi-Objective Polynomial Optimization

Finite Convergence and Sharp Minima for Quasi-Equilibrium Problems

Dual Kuhn-Tucker necessary conditions for strict minima of order two for nonsmooth vector optimization problems with inequality and equality constraints

Optimality conditions for robust weakly efficient solutions in uncertain optimization

A Universal Class of Sharpness-Aware Minimization Algorithms

Optimization on Pareto sets: On a theory of multi-objective optimization

Variable Metric Method for Unconstrained Multiobjective Optimization Problems

Existence of Weak Pareto Efficient Solutions of a Vector Optimization Problem under a Closed Constraint Set

The Minimization of Piecewise Functions: Pseudo Stationarity

Cardinality-Constrained Multi-Objective Optimization: Novel Optimality Conditions and Algorithms

Near Optimal Stochastic Algorithms for Finite-Sum Unbalanced Convex-Concave Minimax Optimization

Critical Influence of Overparameterization on Sharpness-aware Minimization