A more robust boosting algorithm

Yoav Freund

DOI: https://doi.org/10.48550/arXiv.0905.2138

IF: 5.414

2009-05-13

Machine Learning

Abstract:We present a new boosting algorithm, motivated by the large margins theory for boosting. We give experimental evidence that the new algorithm is significantly more robust against label noise than existing boosting algorithm.

What problem does this paper attempt to address?

Boosting in the presence of label noise

Jakramate Bootkrajang,Ata Kaban

DOI: https://doi.org/10.48550/arXiv.1309.6818

IF: 5.414

2013-09-26

Machine Learning

Abstract:Boosting is known to be sensitive to label noise. We studied two approaches to improve AdaBoost's robustness against labelling errors. One is to employ a label-noise robust classifier as a base learner, while the other is to modify the AdaBoost algorithm to be more robust. Empirical evaluation shows that a committee of robust classifiers, although converges faster than non label-noise aware AdaBoost, is still susceptible to label noise. However, pairing it with the new robust Boosting algorithm we propose here results in a more resilient algorithm under mislabelling.
Boosting the Partial Least Square Algorithm for Regression Modelling

Ling Yu,Tiejun Wu

DOI: https://doi.org/10.1007/s11768-006-5231-z

2006-01-01

Abstract:Boosting algorithms are a class of general methods used to improve the general performance of regression analysis. The main idea is to maintain a distribution over the train set. In order to use the given distribution directly, a modified PLS algorithm is proposed and used as the base learner to deal with the nonlinear multivariate regression problems. Experiments on gasoline octane number prediction demonstrate that boosting the modified PLS algorithm has better general performance over the PLS algorithm.
Experiments with a New Boosting Algorithm

Y. Freund,R. Schapire

1996-07-03

Abstract:In an earlier paper, we introduced a new "boosting" algorithm called AdaBoost which, theoretically, can be used to significantly reduce the error of any learning algorithm that con- sistently generates classifiers whose performance is a little better than random guessing. We also introduced the related notion of a "pseudo-loss" which is a method for forcing a learning algorithm of multi-label concepts to concentrate on the labels that are hardest to discriminate. In this paper, we describe experiments we carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems. We performed two sets of experiments. The first set compared boosting to Breiman's "bagging" method when used to aggregate various classifiers (including decision trees and single attribute- value tests). We compared the performance of the two methods on a collection of machine-learning benchmarks. In the second set of experiments, we studied in more detail the performance of boosting using a nearest-neighbor classifier on an OCR problem.

Computer Science
Non-Convex Boosting Overcomes Random Label Noise

Sunsern Cheamanunkul,Evan Ettinger,Yoav Freund

DOI: https://doi.org/10.48550/arXiv.1409.2905

IF: 5.414

2014-09-09

Machine Learning

Abstract:The sensitivity of Adaboost to random label noise is a well-studied problem. LogitBoost, BrownBoost and RobustBoost are boosting algorithms claimed to be less sensitive to noise than AdaBoost. We present the results of experiments evaluating these algorithms on both synthetic and real datasets. We compare the performance on each of datasets when the labels are corrupted by different levels of independent label noise. In presence of random label noise, we found that BrownBoost and RobustBoost perform significantly better than AdaBoost and LogitBoost, while the difference between each pair of algorithms is insignificant. We provide an explanation for the difference based on the margin distributions of the algorithms.
Boosting Barely Robust Learners: A New Perspective on Adversarial Robustness

Avrim Blum,Omar Montasser,Greg Shakhnarovich,Hongyang Zhang

DOI: https://doi.org/10.48550/arXiv.2202.05920

2022-02-12

Abstract:We present an oracle-efficient algorithm for boosting the adversarial robustness of barely robust learners. Barely robust learning algorithms learn predictors that are adversarially robust only on a small fraction $\beta \ll 1$ of the data distribution. Our proposed notion of barely robust learning requires robustness with respect to a "larger" perturbation set; which we show is necessary for strongly robust learning, and that weaker relaxations are not sufficient for strongly robust learning. Our results reveal a qualitative and quantitative equivalence between two seemingly unrelated problems: strongly robust learning and barely robust learning.

Machine Learning
Boosting The Pls Algorithm For Regressive Modelling

Ling Yu,Tiejun Wu

DOI: https://doi.org/10.1109/WCICA.2006.1713302

2006-01-01

Abstract:Boosting algorithms are a class of general methods used to improve the generalization performance of regression analysis. The main idea is to maintain a distribution over the train set. In order to use the given distribution directly, a modified PLS algorithm is proposed and used as the base learner to deal with regression problems. Experiments on gasoline octane number prediction demonstrate that boosting the modified PLS algorithm has better generalization performance over the PLS algorithm.
Analysis of NIR Using Boosting Algorithm

乐斌,吴铁军,方骏

DOI: https://doi.org/10.3969/j.issn.1007-7324.2004.01.010

2004-01-01

Abstract:Boosting is a new machine learning algorithm,which is usually utilized to improve the performance of an ensemble of regression algorithms.A boosting regression algorithm using RBF neural networks as a base learner is proposed.Meanwhile this algorithm is applied in the prediction of gasoline octane number.In comparison with other traditional techniques,such as PLS method,MLR method and single RBF neural network,it shows that this method features high learning speed,good approximation and excellent generalization ability.
Further Results on the Margin Explanation of Boosting: New Algorithm and Experiments

LiWei Wang,XiaoCheng Deng,ZhaoXiang Jing,JuFu Feng

DOI: https://doi.org/10.1007/s11432-012-4602-y

2012-01-01

Science China Information Sciences

Abstract:Understanding the empirical success of boosting algorithms is an important theoretical problem in machine learning. One of the most influential works is the margin theory, which provides a series of upper bounds for the generalization error of any voting classifier in terms of the margins of the training data. Recently an equilibrium margin (Emargin) bound which is sharper than previously well-known margin bounds is proposed. In this paper, we conduct extensive experiments to test the Emargin theory. Specifically, we develop an efficient algorithm that, given a boosting classifier (or a voting classifier in general), learns a new voting classifier which usually has a smaller Emargin bound. We then compare the performances of the two classifiers and find that the new classifier often has smaller test errors, which agrees with what the Emargin theory predicts.
Improved Boosting Algorithm With Adaptive Filtration

Yunlong Gao,Feng Gao,Xiaohong Guan

DOI: https://doi.org/10.1109/WCICA.2010.5553968

2010-01-01

Abstract:AdaBoost is known as an effective method to improve the performance of base classifiers both theoretically and empirically. However, previous studies have shown that AdaBoost is always prone to overfitting especially in noisy case. In addition, most current works on Boosting assume that the loss function is fixed and therefore do not take the distinction between noisy case and noise-free case into consideration. In this paper, an improved Boosting algorithm with adaptive filtration is proposed. A filtering algorithm is designed firstly based on Hoeffding Inequality to identify mislabeled or atypical samples. By introducing the filtering algorithm, we manage to modify the loss function such that influences of mislabeled or atypical samples are penalized. Experiments performed on eight different UCI data sets show that the new Boosting algorithm almost always obtains considerably better classification accuracy than AdaBoost. Furthermore, experiments on data with artificially controlled noise indicate that the new Boosting algorithm is more robust to noise than AdaBoost.
Boosting in the presence of outliers: adaptive classification with non-convex loss functions

Alexander Hanbo Li,Jelena Bradic

DOI: https://doi.org/10.1080/01621459.2016.1273116

2015-10-05

Abstract:This paper examines the role and efficiency of the non-convex loss functions for binary classification problems. In particular, we investigate how to design a simple and effective boosting algorithm that is robust to the outliers in the data. The analysis of the role of a particular non-convex loss for prediction accuracy varies depending on the diminishing tail properties of the gradient of the loss -- the ability of the loss to efficiently adapt to the outlying data, the local convex properties of the loss and the proportion of the contaminated data. In order to use these properties efficiently, we propose a new family of non-convex losses named $\gamma$-robust losses. Moreover, we present a new boosting framework, {\it Arch Boost}, designed for augmenting the existing work such that its corresponding classification algorithm is significantly more adaptable to the unknown data contamination. Along with the Arch Boosting framework, the non-convex losses lead to the new class of boosting algorithms, named adaptive, robust, boosting (ARB). Furthermore, we present theoretical examples that demonstrate the robustness properties of the proposed algorithms. In particular, we develop a new breakdown point analysis and a new influence function analysis that demonstrate gains in robustness. Moreover, we present new theoretical results, based only on local curvatures, which may be used to establish statistical and optimization properties of the proposed Arch boosting algorithms with highly non-convex loss functions. Extensive numerical calculations are used to illustrate these theoretical properties and reveal advantages over the existing boosting methods when data exhibits a number of outliers.

Machine Learning,Artificial Intelligence,Statistics Theory,Methodology
Robust boosting for regression problems

Xiaomeng Ju,Matías Salibián-Barrera

DOI: https://doi.org/10.1016/j.csda.2020.107065

2021-01-01

Abstract:<p>Gradient boosting algorithms construct a regression predictor using a linear combination of "base learners". Boosting also offers an approach to obtaining robust non-parametric regression estimators that are scalable to applications with many explanatory variables. The robust boosting algorithm is based on a two-stage approach, similar to what is done for robust linear regression: it first minimizes a robust residual scale estimator, and then improves it by optimizing a bounded loss function. Unlike previous robust boosting proposals this approach does not require computing an ad-hoc residual scale estimator in each boosting iteration. Since the loss functions involved in this robust boosting algorithm are typically non-convex, a reliable initialization step is required, such as an <span class="math"><math>L1</math></span> regression tree, which is also fast to compute. A robust variable importance measure can also be calculated via a permutation procedure. Thorough simulation studies and several data analyses show that, when no atypical observations are present, the robust boosting approach works as well as the standard gradient boosting with a squared loss. Furthermore, when the data contain outliers, the robust boosting estimator outperforms the alternatives in terms of prediction error and variable selection accuracy.</p>

statistics & probability,computer science, interdisciplinary applications
Optimal Minimal Margin Maximization with Boosting

Allan Grønlund,Kasper Green Larsen,Alexander Mathiasen

DOI: https://doi.org/10.48550/arXiv.1901.10789

2019-01-30

Abstract:Boosting algorithms produce a classifier by iteratively combining base hypotheses. It has been observed experimentally that the generalization error keeps improving even after achieving zero training error. One popular explanation attributes this to improvements in margins. A common goal in a long line of research, is to maximize the smallest margin using as few base hypotheses as possible, culminating with the AdaBoostV algorithm by (R{ä}tsch and Warmuth [JMLR'04]). The AdaBoostV algorithm was later conjectured to yield an optimal trade-off between number of hypotheses trained and the minimal margin over all training points (Nie et al. [JMLR'13]). Our main contribution is a new algorithm refuting this conjecture. Furthermore, we prove a lower bound which implies that our new algorithm is optimal.

Machine Learning,Data Structures and Algorithms
SPLBoost: An Improved Robust Boosting Algorithm Based on Self-Paced Learning

Kaidong Wang,Yao Wang,Qian Zhao,Deyu Meng,Xiuwu Liao,Zongben Xu

DOI: https://doi.org/10.1109/tcyb.2019.2957101

IF: 11.8

2021-03-01

IEEE Transactions on Cybernetics

Abstract:It is known that boosting can be interpreted as an optimization technique to minimize an underlying loss function. Specifically, the underlying loss being minimized by the traditional AdaBoost is the exponential loss, which proves to be very sensitive to random noise/outliers. Therefore, several boosting algorithms, e.g., LogitBoost and SavageBoost, have been proposed to improve the robustness of AdaBoost by replacing the exponential loss with some designed robust loss functions. In this article, we present a new way to robustify AdaBoost, that is, incorporating the robust learning idea of self-paced learning (SPL) into the boosting framework. Specifically, we design a new robust boosting algorithm based on the SPL regime, that is, SPLBoost, which can be easily implemented by slightly modifying off-the-shelf boosting packages. Extensive experiments and a theoretical characterization are also carried out to illustrate the merits of the proposed SPLBoost.

automation & control systems,computer science, cybernetics, artificial intelligence
A Bayesian Boosting Model

Alexander Lorbert,David M. Blei,Robert E. Schapire,Peter J. Ramadge

DOI: https://doi.org/10.48550/arXiv.1209.1996

IF: 5.414

2012-09-10

Machine Learning

Abstract:We offer a novel view of AdaBoost in a statistical setting. We propose a Bayesian model for binary classification in which label noise is modeled hierarchically. Using variational inference to optimize a dynamic evidence lower bound, we derive a new boosting-like algorithm called VIBoost. We show its close connections to AdaBoost and give experimental results from four datasets.
On the Margin Explanation of Boosting Algorithms.

Liwei Wang,Masashi Sugiyama,Cheng Yang,Zhi-Hua Zhou,Jufu Feng

2008-01-01

Abstract:Much attention has been paid to the theoretical explanation of the empirical success of AdaBoost. The most influential work is the margin theory, which is essentially an upper bound for the generalization error of any voting classifier in terms of the margin distribution over the training data. However, Breiman raised important questions about the margin explanation by developing a boosting algorithm arc-gv that provably generates a larger minimum margin than AdaBoost. He also gave a sharper bound in terms of the minimum margin, and argued that the minimum margin governs the generalization. In experiments however, arc-gv usually performs worse than AdaBoost, putting the margin explanation into serious doubts. In this paper, we try to give a complete answer to Breiman’s critique by proving a bound in terms of a new margin measure called Equilibrium margin (Emargin). The Emargin bound is uniformly sharper than Breiman’s minimum margin bound. This result suggests that the minimum margin is not crucial for the generalization error. We also show that a large Emargin implies good generalization. Experimental results on benchmark datasets demonstrate that AdaBoost usually has a larger Emargin and a smaller test error than arc-gv, which agrees well with our theory.
Online coordinate boosting

Raphael Pelossof,Michael Jones,Ilia Vovsha,Cynthia Rudin

DOI: https://doi.org/10.1109/iccvw.2009.5457454

2009-09-01

Abstract:We present a new online boosting algorithm for updating the weights of a boosted classifier, which yields a closer approximation to the edges found by Freund and Schapire's AdaBoost algorithm than previous online boosting algorithms. We contribute a new way of deriving the online algorithm that ties together previous online boosting work. The online algorithm is derived by minimizing AdaBoost's loss as a single example is added to the training set. The equations show that the optimization is computationally expensive. However, a fast online approximation is possible. We compare approximation error to edges found by batch AdaBoost on synthetic datasets and generalization error on face datasets and the MNIST dataset.
Optimal and Adaptive Algorithms for Online Boosting

Alina Beygelzimer,Satyen Kale,Haipeng Luo

DOI: https://doi.org/10.48550/arXiv.1502.02651

2015-02-10

Abstract:We study online boosting, the task of converting any weak online learner into a strong online learner. Based on a novel and natural definition of weak online learnability, we develop two online boosting algorithms. The first algorithm is an online version of boost-by-majority. By proving a matching lower bound, we show that this algorithm is essentially optimal in terms of the number of weak learners and the sample complexity needed to achieve a specified accuracy. This optimal algorithm is not adaptive however. Using tools from online loss minimization, we derive an adaptive online boosting algorithm that is also parameter-free, but not optimal. Both algorithms work with base learners that can handle example importance weights directly, as well as by rejection sampling examples with probability defined by the booster. Results are complemented with an extensive experimental study.

Machine Learning
A Study on the Noise Label Influence in Boosting Algorithms: AdaBoost, GBM and XGBoost

Anabel Gómez-Ríos,Julián Luengo,Francisco Herrera

DOI: https://doi.org/10.1007/978-3-319-59650-1_23

2017-01-01

Abstract:In classification, class noise alludes to incorrect labelling of instances and it causes the classifiers to perform worse. In this contribution, we test the resistance against noise of the most influential boosting algorithms. We explain the fundamentals of these state-of-the-art algorithms, providing an unified notation to facilitate their comparison. We analyse how they carry out the classification, what loss functions use and what techniques employ under the boosting scheme.
Boosting Through Optimization of Margin Distributions

Chunhua Shen,Hanxi Li

DOI: https://doi.org/10.1109/tnn.2010.2040484

2010-01-01

IEEE Transactions on Neural Networks

Abstract:Boosting has been of great interest recently in the machine learning community because of the impressive performance for classification and regression problems. The success of boosting algorithms may be interpreted in terms of the margin theory. Recently, it has been shown that generalization error of classifiers can be obtained by explicitly taking the margin distribution of the training data into account. Most of the current boosting algorithms in practice usually optimize a convex loss function and do not make use of the margin distribution. In this brief, we design a new boosting algorithm, termed margin-distribution boosting (MDBoost), which directly maximizes the average margin and minimizes the margin variance at the same time. This way the margin distribution is optimized. A totally corrective optimization algorithm based on column generation is proposed to implement MDBoost. Experiments on various data sets show that MDBoost outperforms AdaBoost and LPBoost in most cases.
A Refined Margin Analysis for Boosting Algorithms Via Equilibrium Margin

Liwei Wang,Masashi Sugiyama,Zhaoxiang Jing,Cheng Yang,Zhi-Hua Zhou,Jufu Feng

DOI: https://doi.org/10.5555/1953048.2021058

2011-01-01

Journal of Machine Learning Research

Abstract:Much attention has been paid to the theoretical explanation of the empirical success of AdaBoost. The most influential work is the margin theory, which is essentially an upper bound for the generalization error of any voting classifier in terms of the margin distribution over the training data. However, important questions were raised about the margin explanation. Breiman (1999) proved a bound in terms of the minimum margin, which is sharper than the margin distribution bound. He argued that the minimum margin would be better in predicting the generalization error. Grove and Schuurmans (1998) developed an algorithm called LP-AdaBoost which maximizes the minimum margin while keeping all other factors the same as AdaBoost. In experiments however, LP-AdaBoost usually performs worse than AdaBoost, putting the margin explanation into serious doubt. In this paper, we make a refined analysis of the margin theory. We prove a bound in terms of a new margin measure called the Equilibrium margin (Emargin). The Emargin bound is uniformly sharper than Breiman's minimum margin bound. Thus our result suggests that the minimum margin may be not crucial for the generalization error. We also show that a large Emargin and a small empirical error at Emargin imply a smaller bound of the generalization error. Experimental results on benchmark data sets demonstrate that AdaBoost usually has a larger Emargin and a smaller test error than LP-AdaBoost, which agrees well with our theory.

A more robust boosting algorithm

Boosting in the presence of label noise

Boosting the Partial Least Square Algorithm for Regression Modelling

Experiments with a New Boosting Algorithm

Non-Convex Boosting Overcomes Random Label Noise

Boosting Barely Robust Learners: A New Perspective on Adversarial Robustness

Boosting The Pls Algorithm For Regressive Modelling

Analysis of NIR Using Boosting Algorithm

Further Results on the Margin Explanation of Boosting: New Algorithm and Experiments

Improved Boosting Algorithm With Adaptive Filtration

Boosting in the presence of outliers: adaptive classification with non-convex loss functions

Robust boosting for regression problems

Optimal Minimal Margin Maximization with Boosting

SPLBoost: An Improved Robust Boosting Algorithm Based on Self-Paced Learning

A Bayesian Boosting Model

On the Margin Explanation of Boosting Algorithms.

Online coordinate boosting

Optimal and Adaptive Algorithms for Online Boosting

A Study on the Noise Label Influence in Boosting Algorithms: AdaBoost, GBM and XGBoost

Boosting Through Optimization of Margin Distributions

A Refined Margin Analysis for Boosting Algorithms Via Equilibrium Margin