Abstract:Margin enlargement over training data has been an important strategy since perceptrons in machine learning for the purpose of boosting the robustness of classifiers toward a good generalization ability. Yet Breiman (1999) showed a dilemma that a uniform improvement on margin distribution does NOT necessarily reduces generalization errors. In this paper, we revisit Breiman's dilemma in deep neural networks with recently proposed spectrally normalized margins, from a novel perspective based on phase transitions of normalized margin distributions in training dynamics. Normalized margin distribution of a classifier over the data, can be divided into two parts: low/small margins such as some negative margins for misclassified samples vs. high/large margins for high confident correctly classified samples, that often behave differently during the training process. Low margins for training and test datasets are often effectively reduced in training, along with reductions of training and test errors; while high margins may exhibit different dynamics, reflecting the trade-off between expressive power of models and complexity of data. When data complexity is comparable to the model expressiveness, high margin distributions for both training and test data undergo similar decrease-increase phase transitions during training. In such cases, one can predict the trend of generalization or test error by margin-based generalization bounds with restricted Rademacher complexities, shown in two ways in this paper with early stopping time exploiting such phase transitions. On the other hand, over-expressive models may have both low and high training margins undergoing uniform improvements, with a distinct phase transition in test margin dynamics. This reconfirms the Breiman's dilemma associated with overparameterized neural networks where margins fail to predict overfitting.

Optimal Margin Distribution Network

Improving generalization of deep neural networks by leveraging margin distribution

On the Optimization of Margin Distribution

Margin Distribution Analysis

Large margin deep neural networks: Theory and algorithms

Large Margin Deep Neural Networks: Theory and Algorithms.

On margin-based generalization prediction in deep neural networks

Distribution of Classification Margins: Are All Data Equal?

Effects of the Optimisation of the Margin Distribution on Generalisation in Deep Architectures

Adversarial Margin Maximization Networks

Robust Large Margin Deep Neural Networks

Scalable Optimal Margin Distribution Machine

On the doubt about margin explanation of boosting

Rethinking Breiman's Dilemma in Neural Networks: Phase Transitions of Margin Dynamics

Boosting Through Optimization of Margin Distributions

Optimal Margin Distribution Learning in Dynamic Environments

Boosting and margin theory

Optimal Margin Distribution Machine with Sparsity Inducing Penalty

A Margin-based Multiclass Generalization Bound via Geometric Complexity

Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin

Learning Towards The Largest Margins.