Abstract:Margin enlargement over training data has been an important strategy since perceptrons in machine learning for the purpose of boosting the robustness of classifiers toward a good generalization ability. Yet Breiman (1999) showed a dilemma that a uniform improvement on margin distribution does NOT necessarily reduces generalization errors. In this paper, we revisit Breiman's dilemma in deep neural networks with recently proposed spectrally normalized margins, from a novel perspective based on phase transitions of normalized margin distributions in training dynamics. Normalized margin distribution of a classifier over the data, can be divided into two parts: low/small margins such as some negative margins for misclassified samples vs. high/large margins for high confident correctly classified samples, that often behave differently during the training process. Low margins for training and test datasets are often effectively reduced in training, along with reductions of training and test errors; while high margins may exhibit different dynamics, reflecting the trade-off between expressive power of models and complexity of data. When data complexity is comparable to the model expressiveness, high margin distributions for both training and test data undergo similar decrease-increase phase transitions during training. In such cases, one can predict the trend of generalization or test error by margin-based generalization bounds with restricted Rademacher complexities, shown in two ways in this paper with early stopping time exploiting such phase transitions. On the other hand, over-expressive models may have both low and high training margins undergoing uniform improvements, with a distinct phase transition in test margin dynamics. This reconfirms the Breiman's dilemma associated with overparameterized neural networks where margins fail to predict overfitting.

On-line learning through simple perceptron with a margin

Statistical Mechanics of On-line Ensemble Teacher Learning through a Novel Perceptron Learning Rule

The Perceptron with Dynamic Margin

Max-margin Based Bayesian Classifier.

Generalization ability of a perceptron with non-monotonic transfer function

Nonlinear Online Classification Algorithm with Probability Margin

On Accelerated Perceptrons and Beyond

Ensemble learning of linear perceptron; Online learning theory

Perceptron Mistake Bounds

A simpler unified analysis of budget perceptrons

Generalization Errors of the Simple Perceptron

On the Perceptron's Compression

On Ψ-Learning

Perceptron like Algorithms for Online Learning to Rank

A Discussion of the Learning in a Simple Perceptron

Adaptive margin support vector machines

Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin

Learning in A Higher-Order Simple Perceptron

Complex-valued Online Classifier

Rethinking Breiman's Dilemma in Neural Networks: Phase Transitions of Margin Dynamics

Large scale analysis of generalization error in learning using margin based classification methods