Abstract:Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. Despite the practical success, the mechanisms behind SAM's generalization enhancements remain elusive, limiting its progress in deep learning optimization. In this work, we investigate SAM's core components for generalization improvement and introduce "Friendly-SAM" (F-SAM) to further enhance SAM's generalization. Our investigation reveals the key role of batch-specific stochastic gradient noise within the adversarial perturbation, i.e., the current minibatch gradient, which significantly influences SAM's generalization performance. By decomposing the adversarial perturbation in SAM into full gradient and stochastic gradient noise components, we discover that relying solely on the full gradient component degrades generalization while excluding it leads to improved performance. The possible reason lies in the full gradient component's increase in sharpness loss for the entire dataset, creating inconsistencies with the subsequent sharpness minimization step solely on the current minibatch data. Inspired by these insights, F-SAM aims to mitigate the negative effects of the full gradient component. It removes the full gradient estimated by an exponentially moving average (EMA) of historical stochastic gradients, and then leverages stochastic gradient noise for improved generalization. Moreover, we provide theoretical validation for the EMA approximation and prove the convergence of F-SAM on non-convex problems. Extensive experiments demonstrate the superior generalization performance and robustness of F-SAM over vanilla SAM. Code is available at

Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimization

Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization

Towards Efficient and Scalable Sharpness-Aware Minimization

Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy

Friendly Sharpness-Aware Minimization

An Adaptive Policy to Employ Sharpness-Aware Minimization

SAMPa: Sharpness-aware Minimization Parallelized

GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization

Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

Provable Sharpness-Aware Minimization with Adaptive Learning Rate

Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training

AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks

Momentum-SAM: Sharpness Aware Minimization without Computational Overhead

Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning

Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate

Towards Understanding Sharpness-Aware Minimization

Fundamental Convergence Analysis of Sharpness-Aware Minimization

How Sharpness-Aware Minimization Minimizes Sharpness?

Why Does Sharpness-Aware Minimization Generalize Better Than SGD?

Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer

Decentralized Stochastic Sharpness-Aware Minimization Algorithm