Abstract:Margin enlargement over training data has been an important strategy since perceptrons in machine learning for the purpose of boosting the robustness of classifiers toward a good generalization ability. Yet Breiman (1999) showed a dilemma that a uniform improvement on margin distribution does NOT necessarily reduces generalization errors. In this paper, we revisit Breiman's dilemma in deep neural networks with recently proposed spectrally normalized margins, from a novel perspective based on phase transitions of normalized margin distributions in training dynamics. Normalized margin distribution of a classifier over the data, can be divided into two parts: low/small margins such as some negative margins for misclassified samples vs. high/large margins for high confident correctly classified samples, that often behave differently during the training process. Low margins for training and test datasets are often effectively reduced in training, along with reductions of training and test errors; while high margins may exhibit different dynamics, reflecting the trade-off between expressive power of models and complexity of data. When data complexity is comparable to the model expressiveness, high margin distributions for both training and test data undergo similar decrease-increase phase transitions during training. In such cases, one can predict the trend of generalization or test error by margin-based generalization bounds with restricted Rademacher complexities, shown in two ways in this paper with early stopping time exploiting such phase transitions. On the other hand, over-expressive models may have both low and high training margins undergoing uniform improvements, with a distinct phase transition in test margin dynamics. This reconfirms the Breiman's dilemma associated with overparameterized neural networks where margins fail to predict overfitting.

The Perceptron with Dynamic Margin

On-line learning through simple perceptron with a margin

On Accelerated Perceptrons and Beyond

A kernel-based Perceptron with dynamic memory

Perceptron Mistake Bounds

Max-margin Basedbayesian Classifier

On the Perceptron's Compression

On the Atypical Solutions of the Symmetric Binary Perceptron

Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin

Pairwise Margin Maximization for Deep Neural Networks

Rethinking Breiman's Dilemma in Neural Networks: Phase Transitions of Margin Dynamics

SoftDoubleMaxMinOver: Perceptron-Like Training of Support Vector Machines

PUMA: margin-based data pruning

Discrepancy Algorithms for the Binary Perceptron

A note on the capacity of the binary perceptron

A Multiclass Kernel Perceptron Algorithm

Nonlinear Online Classification Algorithm with Probability Margin

Critical Window of The Symmetric Perceptron

Generalization ability of a perceptron with non-monotonic transfer function

Large Margin Deep Neural Networks: Theory and Algorithms.