Abstract:Margin theory provides one of the most popular explanations to the success of AdaBoost, where the central point lies in the recognition that margin is the key for characterizing the performance of AdaBoost. This theory has been very influential, e.g., it has been used to argue that AdaBoost usually does not overfit since it tends to enlarge the margin even after the training error reaches zero. Previously the minimum margin bound was established for AdaBoost, however, Breiman (1999) [9] pointed out that maximizing the minimum margin does not necessarily lead to a better generalization. Later, Reyzin and Schapire (2006) [37] emphasized that the margin distribution rather than minimum margin is crucial to the performance of AdaBoost. In this paper, we first present the kth margin bound and further study on its relationship to previous work such as the minimum margin bound and Emargin bound. Then, we improve the previous empirical Bernstein bounds (Audibert et al. 2009; Maurer and Pontil, 2009) [2,30], and based on such findings, we defend the margin-based explanation against Breimanʼs doubts by proving a new generalization error bound that considers exactly the same factors as Schapire et al. (1998) [39] but is sharper than Breimanʼs (1999) [9] minimum margin bound. By incorporating factors such as average margin and variance, we present a generalization error bound that is heavily related to the whole margin distribution. We also provide margin distribution bounds for generalization error of voting classifiers in finite VC-dimension space.

Global Wasserstein Margin Maximization for Boosting Generalization in Adversarial Training

Towards Desirable Decision Boundary by Moderate-Margin Adversarial Training

GAAT: Group Adaptive Adversarial Training to Improve the Trade-Off Between Robustness and Accuracy

Boosting Through Optimization of Margin Distributions

Improving Adversarial Robustness of CNNs Via Maximum Margin

On the doubt about margin explanation of boosting

Margin Distribution Controlled Boosting

Adversarial Margin Maximization Networks

Boosting and margin theory

On the margin explanation of boosting algorithms.

Feature Augmentation for Adversarial Robustness

Faster Margin Maximization Rates for Generic and Adversarially Robust Optimization Methods

Improving generalization of deep neural networks by leveraging margin distribution

Pareto Adversarial Robustness: Balancing Spatial Robustness and Sensitivity-based Robustness

MMA Training: Direct Input Space Margin Maximization through Adversarial Training

Optimal Minimal Margin Maximization with Boosting

On the Optimization of Margin Distribution

Wassertrain: an Adversarial Training Framework Against Wasserstein Adversarial Attacks

Universal generalization guarantees for Wasserstein distributionally robust models