A local boosting algorithm for solving classification problems

ZhangChun-Xia,ZhangJiang-She

IF: 2.035

2008-01-01

Computational Statistics & Data Analysis

Abstract:Based on the boosting-by-resampling version of Adaboost, a local boosting algorithm for dealing with classification tasks is proposed in this paper. Its main idea is that in each iteration, a local...

What problem does this paper attempt to address?

Imbalanced Data Sets Classification Method Based on Over-Sampling Technique

WANG Chunyu,SU Hongye,QU Yu,CHU Jian

DOI: https://doi.org/10.3778/j.issn.1002-8331.2011.01.038

2011-01-01

Computer Engineering and Applications Journal

Abstract:Classification of data with imbalanced class distribution is a research focus on machine learning.In order to resolve the imbalanced problems,especially those of the poor predictive accuracy over the minority class,this paper presents an improved approach,AdaBoost-SVM-OBMS,which is based on a combination of Boosting,an ensemble-based learning algorithm,and an improved over-sampling method based on misclassified samples.In this approach,using support vector machine as base classifier,the misclassified samples are identified during each iteration.Subsequently,they are used to separately generate new samples for the majority and minority classes.The new samples are then added to the original training set to retrain the classification model,which is used to improve the prediction of hard samples.This method is evaluated,in terms of the AUC,F-value,and G-mean,on eight imbalanced data sets.Results indicate that the improved approach produces high prediction in imbalanced data sets.
Multi-class AdaBoost ELM

yefeng shen,yunliang jiang,weicong liu,yong liu

DOI: https://doi.org/10.1007/978-3-319-14066-7_18

2015-01-01

Abstract:Extreme learning machine (ELM) is a competitive machine learning technique, which is simple in theory and fast in implementation, it can identify faults quickly and precisely as compared with traditional identification techniques. As verified by the simulation results, ELM tends to have better scalability and can achieve much better generalization performance and much faster learning speed comparing with traditional SVM. In this paper, we introduce a Multi-class AdaBoost based ELM ensemble method. In our approach, the ELM algorithm is selected as the basic ensemble predictor due to its rapid speed and good performance. Compared with the existed boosting ELM algorithm, our algorithm can be directly used in multi-class classification problem. We also carried out comparable experiments with face recognition datasets, the experimental results show that the proposed algorithm can not only make the predicting result more stable, but also achieve better generalization performance.
A Method for Accurate Localization of Facial Features

Zhihui Jiao,Weize Zhang,Ruofeng Tong

DOI: https://doi.org/10.1109/ETCS.2009.585

2009-01-01

Abstract:Facial features localization, as an important technique in human face analyzing, is specialized in searching for facial features (eyes, nose, mouth, etc) within a given region in an image. Although there are many methods to locate the facial features, most of them are computationally expensive. This paper presents a simple method that uses spatial filter for accurate feature detection in face images. When using this algorithm in complex images (images that contain both human faces and some other objects), we find that, there are many false positive candidates. In order to eliminate these candidates, we use a new variant of AdaBoost classifiers based face detection to detect the face regions, thus limit the regions that feature localization algorithm analyzed.
Boosting Local Naïve Bayesian Rules

Zhipeng Xie

DOI: https://doi.org/10.1007/978-3-642-01510-6_87

2009-01-01

Abstract:Several classification algorithms based on local naïve Bayesian rules have been recently developed to provide high predictability. However, most of them use classifier selection strategy in decision making. To make use of classifier fusion strategy, this paper investigates a boosting algorithm for local naïve Bayesian rules. Firstly, we develop an algorithmic framework as a forward stage-wise additive model. Then, a construction algorithm for lazy naïve Bayesian rules is designed to materialize the algorithmic framework. The construction algorithm starts from the most general rule, and uses a greedy search to grow the antecedent repeatedly in order to get a better rule at each step. Experimental results show that the proposed method has successfully reduced the overall error rate on a variety of domains, compared with boosted naïve Bayesian classifier, and lazy Bayesian rule algorithm.
LOCAL MODELS—THE KEY TO BOOSTING STABLE LEARNERS SUCCESSFULLY

Kai Ming Ting,Lian Zhu,Jonathan R. Wells

DOI: https://doi.org/10.1111/j.1467-8640.2012.00441.x

2012-01-01

Computational Intelligence

Abstract:Boosting has been shown to improve the predictive performance of unstable learners such as decision trees, but not of stable learners like Support Vector Machines (SVM), k‐nearest neighbors and Naive Bayes classifiers. In addition to the model stability problem, the high time complexity of some stable learners such as SVM prohibits them from generating multiple models to form an ensemble for large data sets. This paper introduces a simple method that not only enables Boosting to improve the predictive performance of stable learners, but also significantly reduces the computational time to generate an ensemble of stable learners such as SVM for large data sets that would otherwise be infeasible. The method proposes to build local models, instead of global models; and it is the first method, to the best of our knowledge, to solve the two problems in Boosting stable learners at the same time. We implement the method by using a decision tree to define local regions and build a local model for each local region. We show that this implementation of the proposed method enables successful Boosting of three types of stable learners: SVM, k‐nearest neighbors and Naive Bayes classifiers.
Effective Boosting of Naïve Bayesian Classifiers by Local Accuracy Estimation

Zhipeng Xie

DOI: https://doi.org/10.1007/978-3-642-01307-2_88

2009-01-01

Abstract:This paper investigates an effective boosting method for naïve Bayesian classifiers. Existing work has shown that the boosted naïve Bayesian classifier is not so effective in error rate reduction as the boosted decision tree (or boosted decision stump). This phenomenon may be caused by the combination of a couple of facts. To solve the problem, the local accuracies of a naïve Bayesian base classifier should be used to replace the global accuracy (or global error rate) in the traditional boosting methods. Based on the analysis, we propose an effective boosted naïve Bayesian method which uses a C4.5 decision tree as the local-accuracy evaluator for each base classifier. At each round, two classifiers are constructed: one for the naïve Bayesian base classifier, while the other for the C4.5 evaluator. The estimated local accuracy plays an important role, not only in updating the weights of training examples but also in determining the vote weights of base classifiers. Finally, it has been shown by experimental comparison that our method has achieved much lower error rate on average in a set of domains than the AdaBoost.M1 of naïve Bayesian classifiers.
A Weighted-Resampling Based Transfer Learning Algorithm.

Xiaobo Liu,Zhentao Liu,Guangjun Wang,Zhihua Cai,Harry Zhang

DOI: https://doi.org/10.1109/ijcnn.2017.7965853

2017-01-01

Abstract:Transfer learning has attracted more and more attention, and many scholars proposed some useful strategies. Boosting is the main strategy for transfer learning. In boosting, resampling is preferred over reweighting, and it can be applied to any base learner. In this paper, we propose a weighted-resampling method for transfer learning, called TrResampling. Firstly, resampling is applied to the data with heaven weight in the source domain, and the resampled data is used with the target data as the training data to build a classifier. Then the TrAdaBoost algorithm is used to adjust the weights of source data and target data. We discuss Decision Tree, Naive Bayes, and SVM as the base learner in TrResampling, and choose the suitable for TrResampling. In order to illustrate the performance of the proposed algorithm, we compare TrResampling with the state-of-the-art algorithm TrAdaBoost and the base learner Decision Tree, experimental results on UCI data sets indicate that TrResampling is superior to TrAdaBoost and Decision Tree on many data sets.
Boosting Bootstrap Fld Subspaces In Classification

Shen Daoyi,Yu Nenghai,Pang Yanwei,Wang Lei

IF: 1.019

2008-01-01

Chinese Journal of Electronics

Abstract:Propose a new ensemble learning algorithm based on Fisher linear discriminant analysis (FLD or LDA) and Adaboost.M2 for multi-class classification problem. On each iteration of boosting, a number of bootstrap training subsets are first sampled from the original training set according to the sample weights, and FLD is employed in each subset to construct a number of bootstrap FLD subspaces. Then the subspace with the minimum weighted K-nearest neighbor (KNN) pseudo-loss was selected to form the final strong classifier and update the sample weights. The main advantage of our algorithm is that its application is not restricted to the Gaussian assumption of data distribution, and the learning criterion is directly related to the minimum training error. Through the experiments on two real-world data sets, we demonstrate that our algorithm is superior to other techniques, especially for complex data distribution.
Local Boosting for Weakly-Supervised Learning

Rongzhi Zhang,Yue Yu,Jiaming Shen,Xiquan Cui,Chao Zhang

DOI: https://doi.org/10.1145/3580305.3599417

2023-06-05

Abstract:Boosting is a commonly used technique to enhance the performance of a set of base models by combining them into a strong ensemble model. Though widely adopted, boosting is typically used in supervised learning where the data is labeled accurately. However, in weakly supervised learning, where most of the data is labeled through weak and noisy sources, it remains nontrivial to design effective boosting approaches. In this work, we show that the standard implementation of the convex combination of base learners can hardly work due to the presence of noisy labels. Instead, we propose $\textit{LocalBoost}$, a novel framework for weakly-supervised boosting. LocalBoost iteratively boosts the ensemble model from two dimensions, i.e., intra-source and inter-source. The intra-source boosting introduces locality to the base learners and enables each base learner to focus on a particular feature regime by training new base learners on granularity-varying error regions. For the inter-source boosting, we leverage a conditional function to indicate the weak source where the sample is more likely to appear. To account for the weak labels, we further design an estimate-then-modify approach to compute the model weights. Experiments on seven datasets show that our method significantly outperforms vanilla boosting methods and other weakly-supervised methods.

Machine Learning
Two-Stage Multi-Class Adaboost for Facial Expression Recognition

Hongbo Deng,Jianke Zhu,Michael R. Lyu,Irwin King

DOI: https://doi.org/10.1109/ijcnn.2007.4371439

2007-01-01

Abstract:Although AdaBoost has achieved great success, it still suffers from following problems: (1) the training process could be unmanageable when the number of features is extremely large; (2) the same weak classifier may be learned multiple times from a weak classifier pool, which does not provide additional information for updating the model; (3) there is an imbalance between the amount of the positive samples and that of the negative samples for multi-class classification problems. In this paper, we propose a two-stage AdaBoost learning framework to select and fuse the discriminative feature effectively. Moreover, an improved AdaBoost algorithm is developed to select weak classifiers. Instead of boosting in the original feature space, whose dimensionality is usually very high, multiple feature subspaces with lower dimensionality are generated. In the first stage, boosting is carried out in each subspace. Then the trained classifiers are further combined with simple fusion method in the second stage. Experimental results on facial expression recognition data demonstrate that our proposed algorithms not only reduce the computational cost for training, but also achieve comparable classification performance.
Over-sampling Algorithm Based on Adaboost in Unbalanced Data Set

HAN Hui,WANG Wenyuan,MAO Binghuan

DOI: https://doi.org/10.3969/j.issn.1000-3428.2007.10.074

2007-01-01

Abstract:To improve the classification performance of minority class,this paper combines the advantages of boosting and over-sampling,and presents an over-sampling algorithm based on MCMO-Boost of Adaboost.MCMO-Boost is compared with C4.5,Adaboost and SMOTE,and the results show that MCMO-Boost performs better than others for the classification performance of minority class and the whole data set.
Z-Adaboost: Boosting 2-thresholded weak classifiers for object detection

Zhang Weize,Tong Ruofeng,Dong Jinxiang

DOI: https://doi.org/10.1109/IITA.2008.147

2008-01-01

Abstract:This paper presents a new variant of AdaBoost based on Viola and Jones' framework, called Z-AdaBoost. Instead of modifying the mechanism of selecting optimal weak classifiers as other variants do, Z-AdaBoost strengthens the discriminating power of weak classifiers themselves by expanding them into 2-thresholded ones, which guarantees a better classification with smaller error. And a linear online algorithm is adopted to select the optimal values for the 2 thresholds. The idea is simple as the bound on the accuracy of the final hypothesis improves when any of the weak hypotheses is improved. The experimental results under a rigid and objective detection criterion on MIT+CMU upright face test set show that Z-AdaBoost explores some potentially discriminative features that are ignored during the weak learning process in AdaBoost, resulting in a comparably effective detector with much fewer weak classifiers and features. The approach of Z-AdaBoost can be easily incorporated in other boosting algorithms to even improve their performance. © 2008 IEEE.
Optimization of the aggregation in AdaBoost algorithm by particle swarm optimization and its application in classification problems

Shibo Gao,Yuntao Zhang

DOI: https://doi.org/10.1109/ICNC.2010.5583732

2010-01-01

Abstract:In this paper, particle swarm optimization (PSO) algorithm and Adaptive Boosting (AdaBoost) algorithm are combined to form a hybrid learning algorithm PSO-AB to boost the classification ability of support vector machine (SVM). This hybrid adopts SVM to classify the experimental data, uses AdaBoost algorithm to boost the classification results, and then uses PSO to optimize the boosted results. Experimental results of two clinical data show that AdaBoost algorithm could improve the accuracy of training set extremely, but for the testing set the result is not satisfactory. PSO-AB makes it possible to maximize the testing accuracy of AdaBoost algorithm on the premise of that the accuracy of training set is still exact, and will be a more effective method to classification problems compared to AdaBoost algorithm.
LIUBoost : Locality Informed Underboosting for Imbalanced Data Classification

Sajid Ahmed,Farshid Rayhan,Asif Mahbub,Md. Rafsan Jani,Swakkhar Shatabda,Dewan Md. Farid,Chowdhury Mofizur Rahman

DOI: https://doi.org/10.48550/arXiv.1711.05365

2017-11-15

Abstract:The problem of class imbalance along with class-overlapping has become a major issue in the domain of supervised learning. Most supervised learning algorithms assume equal cardinality of the classes under consideration while optimizing the cost function and this assumption does not hold true for imbalanced datasets which results in sub-optimal classification. Therefore, various approaches, such as undersampling, oversampling, cost-sensitive learning and ensemble based methods have been proposed for dealing with imbalanced datasets. However, undersampling suffers from information loss, oversampling suffers from increased runtime and potential overfitting while cost-sensitive methods suffer due to inadequately defined cost assignment schemes. In this paper, we propose a novel boosting based method called LIUBoost. LIUBoost uses under sampling for balancing the datasets in every boosting iteration like RUSBoost while incorporating a cost term for every instance based on their hardness into the weight update formula minimizing the information loss introduced by undersampling. LIUBoost has been extensively evaluated on 18 imbalanced datasets and the results indicate significant improvement over existing best performing method RUSBoost.

Machine Learning
Feature Extraction Algorithm Based on Boosting Bootstrap FLD Projections

沈道义,俞能海,庞彦伟,王雷

2009-01-01

Abstract:In this paper we propose a feature extraction algorithm for two-class problem based on Adaboost algorithm.In this algorithm,we first sample a large number of bootstrap training subsets from the original training set and use FLD in each subset to construct a large number of corresponding one-dimensional(1-d)bootstrap projections.Then at each step of Adaboost algorithm,the bootstrap projection with the minimum weighted 1-Nearest Neighbor(1NN)classification error was selected to combine the final strong classifier.In this manner,our method is able to overcome the limitations of the traditional learning algorithms,such as the classification performance being depending on the data distribution,and the feature extraction rules being not directly related to the training error.Experiment results show that our method is superior to other traditional techniques,especially for the data of complex structure.
Boosting local binary pattern (LBP)-Based face recognition

Guangcheng Zhang,Xiangsheng Huang,Stan Z. Li,Yangsheng Wang,Xihong Wu

DOI: https://doi.org/10.1007/978-3-540-30548-4_21

2004-01-01

Abstract:This paper presents a novel approach for face recognition by boosting statistical local features based classifiers The face image is scanned with a scalable sub-window from which the Local Binary Pattern (LBP) histograms [14] are obtained to describe the local features of a face image The multi-class problem of face recognition is transformed into a two-class one by classifying every two face images as intra-personal or extra-personal ones [9] The Chi square distance between corresponding Local Binary Pattern histograms of two face images is used as discriminative feature for intra/extra-personal classification We use AdaBoost algorithm to learn a similarity of every face image pairs The proposed method was tested on the FERET FA/FB image sets and yielded an exciting recognition rate of 97.9%.
BalancedBoost: A hybrid approach for real-time network traffic classification

Hengyi Wei,Baocheng Sun,Mingming Jing

DOI: https://doi.org/10.1109/icccn.2014.6911833

2014-08-01

Abstract:Real-time network traffic classification suffers from class imbalance and concept drift problem. Moreover, many features for offline analysis becomes unavailable due to the time requirement of giving results as quickly as possible. Several techniques, such as boosting and data sampling, have been used to solve the class imbalance problem. In this paper, we proposed a hybrid approach, called BalancedBoost, to address the class imbalance problem. BalancedBoost is based on a combination of data sampling and boosting algorithm. In order to reduce the impact of traffic dynamics and filter the redundant features, a feature selection algorithm based on symmetrical uncertainty is also presented. We conducted experiments on UNIBS datasets and compared our method to other similar proposed approaches including SMOTEBoost, RUSBoost and OverBagging. Experiment results obtained by our method are promising in terms of F-measure. BalancedBoost is able to improve the classification performance of the minority class. It is simple and efficient.
A Regression Algorithm Based on AdaBoost

Lin Gao,Feng Gao,Xiaohong Guan,Dianmin Zhou,Jie Li

DOI: https://doi.org/10.1109/wcica.2006.1713209

2006-01-01

Abstract:The key to successful machine learning methods is learning quality or accuracy. Boosting has proved to be effective method for improving learning quality of a weak learning algorithm with wide applications to classifications problems. However few successful applications were reported on improving regression quality by boosting. This paper presents a new algorithm for regression based on a boosted support vector machine (SVM) method. A regression problem is first converted to a binary classification problem upon the concept of epsiv-insensitive loss. By applying the idea of AdaBoost algorithm, an optimal classification-plane ensemble is constructed with the converted classification data set. Based on this ensemble a regression estimate function is obtained with equivalence to the original regression problem. The analysis shows that for the regression data set, the number of samples with regression error exceeding epsiv decreased exponentially with the number of boosting iterations. The testing results for an actual data set show that the new algorithm is effective
A scalable direct formulation for multi-class boosting

Sakrapee Paisitkriangkrai,Chunhua Shen,Anton van den Hengel

2012-01-01

Abstract:We present a scalable and effective classification model for multi-class boosting classification. In [1], a direct formulation for multiclass boosting was introduced. Unlike many existing multi-class boosting algorithms, which rely on output coding matrices, the approach in [1] directly maximizes the multi-class margin. The major problem of [1] is its extremely high computational complexity, which hampers its application on real-world problems. In this work, we propose a scalable and simple stage-wise boosting method, which also directly maximizes the multiclass margin. Our approach has several advantages: (1) it is simple and computationally efficient to train. The approach can speed up the training time by more than two orders of magnitude without sacrificing the classification accuracy. (2) Like traditional AdaBoost, it is parameter free and empirically demonstrates excellent generalization performance. In contrast, one has to tune the regularization parameter for the multi-class boosting of [1]. Experimental results on challenging multiclass machine learning and vision tasks demonstrate that the proposed approach substantially improves the convergence rate and accuracy of the final visual detector at no additional computational cost. Our new formulation opens up a new way of incorporating prior knowledge into multi-class problems, e.g., feature sharing, hierarchical features, etc.
A Method to Boost Naïve Bayesian Classifiers

Lili Diao,Keyun Hu,Yuchang Lu,Chunyi Shi

DOI: https://doi.org/10.1007/3-540-47887-6_11

2002-01-01

Abstract:In this paper, we introduce a new method to improve the performance of combining boosting and naïve Bayesian. Instead of combining boosting and Naïve Bayesian learning directly, which was proved to be unsatisfactory to improve performance, we select the training samples dynamically by bootstrap method for the construction of naïve Bayesian classifiers, and hence generate very different or unstable base classifiers for boosting. Besides, we devise a modification for the weight adjusting of boosting algorithm in order to achieve this goal: minimizing the overlapping errors of its constituent classifiers. We conducted series of experiments, which show that the new method not only has performance much better than naïve Bayesian classifiers or directly boosted naïve Bayesian ones, but also much quicker to obtain optimal performance than boosting stumps and boosting decision trees incorporated with naïve Bayesian learning.

A local boosting algorithm for solving classification problems

Imbalanced Data Sets Classification Method Based on Over-Sampling Technique

Multi-class AdaBoost ELM

A Method for Accurate Localization of Facial Features

Boosting Local Naïve Bayesian Rules

LOCAL MODELS—THE KEY TO BOOSTING STABLE LEARNERS SUCCESSFULLY

Effective Boosting of Naïve Bayesian Classifiers by Local Accuracy Estimation

A Weighted-Resampling Based Transfer Learning Algorithm.

Boosting Bootstrap Fld Subspaces In Classification

Local Boosting for Weakly-Supervised Learning

Two-Stage Multi-Class Adaboost for Facial Expression Recognition

Over-sampling Algorithm Based on Adaboost in Unbalanced Data Set

Z-Adaboost: Boosting 2-thresholded weak classifiers for object detection

Optimization of the aggregation in AdaBoost algorithm by particle swarm optimization and its application in classification problems

LIUBoost : Locality Informed Underboosting for Imbalanced Data Classification

Feature Extraction Algorithm Based on Boosting Bootstrap FLD Projections

Boosting local binary pattern (LBP)-Based face recognition

BalancedBoost: A hybrid approach for real-time network traffic classification

A Regression Algorithm Based on AdaBoost

A scalable direct formulation for multi-class boosting

A Method to Boost Naïve Bayesian Classifiers