Abstract:In deep learning, finding flat minima of loss function is a hot research topic in improving generalization. The existing methods usually find flat minima by sharpness minimization algorithms. However, these methods suffer from insufficient flexibility for optimization and generalization due to their ignorance of loss value. This article theoretically and experimentally explores the sharpness minimization algorithms for neural networks. First, a novel scale-invariant sharpness which is called scale-adaptive central moment sharpness (SA-CMS) is proposed. This sharpness is not only scale-invariant but can characterize the nature of loss surface clearly. Based on the proposed sharpness, this article further derives a new regularization term by integrating the different orders of the sharpness. Particularly, a host of sharpness minimization functions such as local entropy can be covered by this regularization term. Then the central moment sharpness generating function is introduced as a new objective function. Moreover, theoretical analyses indicate that the new objective function has a smoother landscape and prefer converging to flat local minima. Furthermore, a computationally efficient two-stage algorithm is developed to minimize the objective function. Compared with other algorithms, the two-stage loss-sharpness minimization (TSLSM) algorithm offers a more flexible optimization target for different training stages. On a variety of learning tasks with both small and large batch sizes, this algorithm is more universal and effective, and meanwhile achieves or surpasses the generalization performance of the state-of-the-art sharpness minimization algorithms.

A smoothing algorithm for training max-min neural networks.

A Convergent Smoothing Algorithm for Training Max -Min Fuzzy Neural Networks.

A Neural Network Model For General Minimax Problem

A Smoothing Trust-Region Newton-CG Method for Minimax Problem

Axiomatization of Gradient Smoothing in Neural Networks

Smoothing Neural Network for Constrained Non-Lipschitz Optimization with Applications

Smooth Min-Max Monotonic Networks

Mini-max Initialization for Function Approximation.

A Modified Gradient Learning Algorithm with Smoothing L-1/2 Regularization for Takagi-Sugeno Fuzzy Models

Neural Network for Nonsmooth, Nonconvex Constrained Minimization Via Smooth Approximation.

The min-max function differentiation and training of fuzzy neural networks

A smoothing interval neural network

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

MinMax Networks

Regularizing Scale-Adaptive Central Moment Sharpness for Neural Networks

A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problems

Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Smoothing Algorithms for Nonsmooth and Nonconvex Minimization over the Stiefel Manifold

A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimiax Optimization

Smoothing Neural Network for L 0 Regularized Optimization Problem with General Convex Constraints