Research on Ensemble Learning

Feng Zhou,Baoliang Lu

2007-01-01

Abstract:1 Background Ensemble Learning Probabilistic Classifier M 3 Framework Summary 2 First Method Problem Description Decomposition Integration Comparison Summary 3 Second Method Problem Revisited Homo-pairwise Combination Comparison

What problem does this paper attempt to address?

Ensemble Methods For Class Imbalance Learning

Xu-Ying Liu,Zhi-Hua Zhou

DOI: https://doi.org/10.1002/9781118646106.ch4

2013-01-01

Abstract:Ensemble learning is an important paradigm in machine learning, which uses a set of classifiers to make predictions. The generalization ability of an ensemble is generally much stronger than that of individual ensemble members. Ensemble learning is widely exploited in the literature of class imbalance learning. This chapter introduces ensemble learning and gives an overview of ensemble methods for class imbalance learning.
Joint Sparse Learning for Classification Ensemble.

Lin Li,Fei Tong,Rustam Stolkin,Jinwen Hu,Feng Yang

DOI: https://doi.org/10.1109/icca.2017.8003205

2017-01-01

Abstract:Ensemble methods use multiple classifiers to achieve better decisions than could be achieved using any of the constituent classifiers alone. However, both theoretical and experimental evidence have shown that very large ensembles are not necessarily superior, and small ensembles can often achieve better results. In this paper, we show how to combine a set of weak classifiers into a robust ensemble by using a joint sparse representation method, which assigns a sparse coefficient vector to the decision of each classifier. The sparse vector contains many zero entries, and thus the final ensemble only employs a small number of classifiers, corresponding to non-zero entries. Training data are partitioned into several sub-groups to generate sub-underdetermined systems. The joint sparse method enables these sub-groups to then share their information about individual classifiers, to obtain an improved overall classification. Partitioning the training dataset into subgroups makes the proposed joint sparse ensemble method parallelizable, making it suitable for large scale problems. In contrast, previous work on sparse approaches to ensemble learning was limited to datasets smaller than the number of classifiers. Two different strategies are described for generating the sub-underdetermined systems, and experiments show these to be effective when tested with two different data manipulation methods. Experiments evaluate the performance of the joint sparse ensemble learning method in comparison to five other state-of-the-art methods from the literature, each designed to train small and efficient ensembles. Results suggest that joint sparse ensemble learning outperforms other algorithms on most datasets.
Comparison among Methods of Ensemble Learning

Shaohua Wan,Hua Yang

DOI: https://doi.org/10.1109/isbast.2013.50

2013-01-01

Abstract:Ensemble learning refers to a collection of methods that learn a target function by training a number of individual learners and combining their predictions. We explore four popular methods (bagging, boosting, stacking and random forest) of combining their outputs, for classification and training time and regression problems. Following this, experimental evaluations are performed on UCI datasets.
Ensemble Transfer Learning Algorithm

Xiaobo Liu,Zhentao Liu,Guangjun Wang,Zhihua Cai,Harry Zhang

DOI: https://doi.org/10.1109/access.2017.2782884

IF: 3.9

2017-01-01

IEEE Access

Abstract:Transfer learning and ensemble learning are the new trends for solving the problem that training data and test data have different distributions. In this paper, we design an ensemble transfer learning framework to improve the classification accuracy when the training data are insufficient. First, a weightedresampling method for transfer learning is proposed, which is named TrResampling. In each iteration, the data with heavy weights in the source domain are resampled, and the TrAdaBoost algorithm is used to adjust the weights of the source data and target data. Second, three classic machine learning algorithms, namely, naive Bayes, decision tree, and SVM, are used as the base learners of TrResampling, where the base learner with the best performance is chosen for transfer learning. To illustrate the performance of TrResampling, the TrAdaBoost and decision tree are used for evaluation and comparison on 15 UCI data sets, TrAdaBoost, ARTL, and SVM are used for evaluation and comparison on five text data sets. According to the experimental results, our proposed TrResampling is superior to the state-of-the-art learning methods on UCI data sets and text data sets. In addition, TrResampling, bagging-based transfer learning algorithm, and MultiBoosting-based transfer learning algorithm (TrMultiBoosting) are assembled in the framework, and we compare the three ensemble transfer learning algorithms with TrAdaBoost to illustrate the framework's effective transfer ability.
Ensemble Methods: Foundations and Algorithms

Zhi-Hua Zhou

2012-01-01

Abstract:An up-to-date, self-contained introduction to a state-of-the-art machine learning approach, Ensemble Methods: Foundations and Algorithms shows how these accurate methods are used in real-world tasks. It gives you the necessary groundwork to carry out further research in this evolving field. After presenting background and terminology, the book covers the main algorithms and theories, including Boosting, Bagging, Random Forest, averaging and voting schemes, the Stacking method, mixture of experts, and diversity measures. It also discusses multiclass extension, noise tolerance, error-ambiguity and bias-variance decompositions, and recent progress in information theoretic diversity. Moving on to more advanced topics, the author explains how to achieve better performance through ensemble pruning and how to generate better clustering results by combining multiple clusterings. In addition, he describes developments of ensemble methods in semi-supervised learning, active learning, cost-sensitive learning, class-imbalance learning, and comprehensibility enhancement.
Exploration of Classification Confidence in Ensemble Learning.

Leijun Li,Qinghua Hu,Xiangqian Wu,Daren Yu

DOI: https://doi.org/10.1016/j.patcog.2014.03.021

IF: 8

2014-01-01

Pattern Recognition

Abstract:Ensemble learning has attracted considerable attention owing to its good generalization performance. The main issues in constructing a powerful ensemble include training a set of diverse and accurate base classifiers, and effectively combining them. Ensemble margin, computed as the difference of the vote numbers received by the correct class and the another class received with the most votes, is widely used to explain the success of ensemble learning. This definition of the ensemble margin does not consider the classification confidence of base classifiers. In this work, we explore the influence of the classification confidence of the base classifiers in ensemble learning and obtain some interesting conclusions. First, we extend the definition of ensemble margin based on the classification confidence of the base classifiers. Then, an optimization objective is designed to compute the weights of the base classifiers by minimizing the margin induced classification loss. Several strategies are tried to utilize the classification confidences and the weights. It is observed that weighted voting based on classification confidence is better than simple voting if all the base classifiers are used. In addition, ensemble pruning can further improve the performance of a weighted voting ensemble. We also compare the proposed fusion technique with some classical algorithms. The experimental results also show the effectiveness of weighted voting with classification confidence.
A New Rotation Forest Ensemble Algorithm

Wen Chenglin,Huai Tingting,Zhang Qinghua,Song Zhihuan,Cao Feilong

DOI: https://doi.org/10.1007/s13042-022-01613-8

2022-01-01

International Journal of Machine Learning and Cybernetics

Abstract:Random forest, a popular ensemble approach in machine learning, has received much attention of researchers in different fields due to its excellent performance. Especially, in the study of classification, it is often used as an effective classifier. Considering that the accuracy and diversity of each base classifier are two main factors that affect the performance of random forest, this paper proposes a new rotation forest ensemble method to increase the diversity of each tree in the forest, which is based on feature extension and transformation. Also, a weighting vote for base classifiers is applied to integrate the final ensemble results instead of to average the accuracy of ensemble learners. In order to illustrate the effectiveness of the proposed algorithm, the experiments conduct with thirty benchmark classification datasets available from the UCI repository and two face recognition databases. Experimental results demonstrate that the proposed algorithm can achieve higher classification accuracy in most cases compared to the other ensemble classifiers.
A Double Penalty Model for Ensemble Learning

Wenjia Wang,Yi-Hui Zhou

DOI: https://doi.org/10.3390/math10234532

IF: 2.4

2022-01-01

Mathematics

Abstract:Modern statistical learning techniques often include learning ensembles, for which the combination of multiple separate prediction procedures (ensemble components) can improve prediction accuracy. Although ensemble approaches are widely used, work remains to improve our understanding of the theoretical underpinnings of aspects such as identifiability and relative convergence rates of the ensemble components. By considering ensemble learning for two learning ensemble components as a double penalty model, we provide a framework to better understand the relative convergence and identifiability of the two components. In addition, with appropriate conditions the framework provides convergence guarantees for a form of residual stacking when iterating between the two components as a cyclic coordinate ascent procedure. We conduct numerical experiments on three synthetic simulations and two real world datasets to illustrate the performance of our approach, and justify our theory.
Classifiers Combination with Observational Learning

Yu Fan,Yang Li-Ying,Qin Zheng

2006-01-01

Abstract:Ensemble method has shown the potential to increase classification accuracy beyond the level reached by an individual classifier alone. Observational Learning Algorithm (OLA) is an ensemble method based on social learning theory. Previous work focused on OLA for homogeneous ensembles, such as neural networks ensembles. In this paper, OLA for heterogeneous ensembles was proposed, which is a process with three steps: training, observing, and retraining. Experiments on five datasets from the UCI repository show that, OLA outperforms the individual base learner and majority voting when base learners are not capable enough for the given task. Bias-variance decomposition of the error indicates that OLA can reduce both bias and variance.
Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [review Article]

Ye Ren,Le Zhang,P. N. Suganthan

DOI: https://doi.org/10.1109/mci.2015.2471235

IF: 9.809

2016-01-01

IEEE Computational Intelligence Magazine

Abstract:Ensemble methods use multiple models to get better performance. Ensemble methods have been used in multiple research fields such as computational intelligence, statistics and machine learning. This paper reviews traditional as well as state-of-the-art ensemble methods and thus can serve as an extensive summary for practitioners and beginners. The ensemble methods are categorized into conventional ensemble methods such as bagging, boosting and random forest, decomposition methods, negative correlation learning methods, multi-objective optimization based ensemble methods, fuzzy ensemble methods, multiple kernel learning ensemble methods and deep learning based ensemble methods. Variations, improvements and typical applications are discussed. Finally this paper gives some recommendations for future research directions.
Ensemble Method for Classification of High-Dimensional Data.

Yongjun Piao,Hyun Woo Park,Cheng Hao Jin,Keun Ho Ryu

DOI: https://doi.org/10.1109/bigcomp.2014.6741445

2014-01-01

Abstract:Ensemble methods, also known as classifier combination were often used to improve the performance of classification. Growing problem of data dimensionality makes a various challenges for supervised learning. Generally used classification methods such as decision tree, neural network and support vector machines were difficult to be directly applied on high-dimensional datasets. In this paper, we proposed an ensemble method for classification of high-dimensional data, with each classifier constructed from a different set of features determined by partition of redundant features. In our method, the redundancy of features was considered to divide the original feature space. Then, each generated feature subset was trained by support vector machine and the results of each classifier were combined by the majority voting method. The efficiency and effectiveness of our method were demonstrated through comparisons with other ensemble techniques, and the results showed that our method outperformed other methods.
Ensemble learning: A survey

Omer Sagi,Lior Rokach

DOI: https://doi.org/10.1002/widm.1249

2018-02-27

WIREs Data Mining and Knowledge Discovery

Abstract:Ensemble methods are considered the state‐of‐the art solution for many machine learning challenges. Such methods improve the predictive performance of a single model by training multiple models and combining their predictions. This paper introduce the concept of ensemble learning, reviews traditional, novel and state‐of‐the‐art ensemble methods and discusses current challenges and trends in the field. This article is categorized under: Algorithmic Development > Ensemble Methods Technologies > Machine Learning Technologies > Classification An ensemble of seven classification trees. Every tree provides each its own classification for the instance x. Using majority voting instance x should be classified as “1” since five trees out of seven vote for “1”.
Clusterer ensemble

Zhi-Hua Zhou,Wei Tang

DOI: https://doi.org/10.1016/j.knosys.2005.11.003

IF: 8.139

2006-01-01

Knowledge-Based Systems

Abstract:Ensemble methods that train multiple learners and then combine their predictions have been shown to be very effective in supervised learning. This paper explores ensemble methods for unsupervised learning. Here, an ensemble comprises multiple clusterers, each of which is trained by k-means algorithm with different initial points. The clusters discovered by different clusterers are aligned, i.e. similar clusters are assigned with the same label, by counting their overlapped data items. Then, four methods are developed to combine the aligned clusterers. Experiments show that clustering performance could be significantly improved by ensemble methods, where utilizing mutual information to select a subset of clusterers for weighted voting is a nice choice. Since the proposed methods work by analyzing the clustering results instead of the internal mechanisms of the component clusterers, they are applicable to diverse kinds of clustering algorithms.
Less Is More: A Comprehensive Framework for the Number of Components of Ensemble Classifiers

Hamed Bonab,Fazli Can

DOI: https://doi.org/10.1109/TNNLS.2018.2886341

Abstract:The number of component classifiers chosen for an ensemble greatly impacts the prediction ability. In this paper, we use a geometric framework for a priori determining the ensemble size, which is applicable to most of the existing batch and online ensemble classifiers. There are only a limited number of studies on the ensemble size examining majority voting (MV) and weighted MV (WMV). Almost all of them are designed for batch-mode, hardly addressing online environments. Big data dimensions and resource limitations, in terms of time and memory, make the determination of ensemble size crucial, especially for online environments. For the MV aggregation rule, our framework proves that the more strong components we add to the ensemble, the more accurate predictions we can achieve. For the WMV aggregation rule, our framework proves the existence of an ideal number of components, which is equal to the number of class labels, with the premise that components are completely independent of each other and strong enough. While giving the exact definition for a strong and independent classifier in the context of an ensemble is a challenging task, our proposed geometric framework provides a theoretical explanation of diversity and its impact on the accuracy of predictions. We conduct a series of experimental evaluations to show the practical value of our theorems and existing challenges.
Ensemble Learning Method for Classification: Integrating Data Envelopment Analysis with Machine Learning

Qingxian An,Siwei Huang,Yuxuan Han,You Zhu

DOI: https://doi.org/10.1016/j.cor.2024.106739

IF: 5.159

2024-01-01

Computers & Operations Research

Abstract:In classification tasks with large sample sets, the use of a single classifier carries the risk of overfitting. To overcome this issue, an ensemble of classifier models has often been shown to outperform the use of a single “best” model. Given the rich variety of classifier models available, the selection of the high-efficiency classifiers for a given task dataset remains an urgent challenge. However, most of the previous classifier selection methods only focus on the measurement of classification output performance without considering the computational cost. This paper proposes a new ensemble learning method to improve the classification quality for big datasets by using data envelopment analysis. It contains the following two stages: classifier selection and classifier combination. In the first stage, the commonly used classifiers are evaluated on the basis of their performance on resource consumption and classification output performance using the range directional model (RDM); then, the most efficient classifiers are selected. In the second stage, the classifier confusion matrix is evaluated using the data envelopment analysis (DEA) cross-efficiency model. Then, the weight for the classifier combination is determined to ensure that classifiers with higher performance have greater weights based on the cross-efficiency values. Experimental results demonstrate the superiority of the cross-efficiency model over the BCC model and the benchmark voting method in model ensemble. Furthermore, our method has been shown to save more computational resources and yields better results than existing methods.
A Comparison of Ensemble Methods in Financial Market Prediction.

Cheng,Wei Xu,Jiajia Wang

DOI: https://doi.org/10.1109/cso.2012.171

2012-01-01

Abstract:Financial time series prediction is always a focus point of researchers and practitioner for its available data and profitability. As recent studies suggest that the employment of ensemble algorithms may improve the performance of a base learner, a compound experiment for comparison of ensemble methods is designed and implemented to investigated the fact that whether the ensemble methods can be employed to improve the performance of the base learner in financial time series prediction. The empirical results suggest that ensemble algorithms are powerful in improving the performances of base learners in financial time series prediction. When compared with Random Subspace and Stacking, Bagging provides a more stable and better improvement. The iteration of ensemble algorithms should be adjusted according to the situation. Higher value of iteration may not always performs well for over fitting may occur.
Classifier Subset Selection Based on Classifier Representation and Clustering Ensemble

Danyang Li,Zhuhong Zhang,Guihua Wen

DOI: https://doi.org/10.1007/s10489-023-04572-x

IF: 5.3

2023-01-01

Applied Intelligence

Abstract:Ensemble pruning can improve the performance and reduce the storage requirements of an integration system. Most ensemble pruning approaches remove low-quality or redundant classifiers by evaluating the classifiers' competence and relationships via their predictions. However, finding the best way to represent classifiers and create ensemble diversity is still a worthy research problem in the ensemble pruning field. To confront this issue, we discuss whether properties other than predictions can represent classifiers and propose a new classifier selection method, classifier-representation- and clustering-ensemble-based ensemble pruning (CRCEEP). In the proposed method, two new classifier-representation-learning methods, local-space- and relative-transformation-based representation, are proposed to obtain more information about classifiers. CRCEEP incorporates the clustering ensemble method to group classifiers and prune redundant learners. Finally, accurate and diverse classifiers are integrated to improve classification performance. Extensive experiments were carried out on UCI datasets, and the experimental results verify CRCEEP's effectiveness and the necessity of classifier representation.
Cocktail Ensemble for Regression

Yang Yu,Zhi-Hua Zhou,Kai Ming Ting

DOI: https://doi.org/10.1109/ICDM.2007.60

2007-01-01

Abstract:This paper is motivated to improve the performance of individual ensembles using a hybrid mechanism in the regression setting. Based on an error-ambiguity decomposition, we formally analyze the optimal linear combination of two base ensembles, which is then extended to multiple individual ensembles via pairwise combinations. The Cocktail ensemble approach is proposed based on this analysis. Experiments over a broad range of data sets show that the proposed approach outperforms the individual ensembles, two other methods of ensemble combination, and two state-of-the-art regression approaches.
A Probabilistic Ensemble Pruning Algorithm

Huanhuan Chen,Peter Tino,Xin Yao

DOI: https://doi.org/10.1109/icdmw.2006.18

2006-01-01

Abstract:An ensemble is a group of learners that work together as a committee to solve a problem. However, the existing ensemble training algorithms sometimes generate unnecessary large ensembles, which consume extra computational resource and may degrade the performance. Ensemble pruning algorithm aims to find a good subset of ensemble members to constitute a small ensemble, which saves the computational resource and performs as well as, or better than, the non-pruned ensemble. This paper introduces a probabilistic ensemble pruning algorithm by choosing a set of "sparse" combination weights, most of which are zero, to prune the large ensemble. In order to obtain the set of sparse combination weights and satisfy the non-negative restriction of the combination weights, a left-truncated, non-negative, Gaussian prior is adopted over every combination weight. Expectation-maximization algorithm is employed to obtain maximum a posterior (MAP) estimation of weight vector. Four benchmark regression problems and another four benchmark classification problems have been employed to demonstrate the effectiveness of the method

Research on Ensemble Learning

Ensemble Methods For Class Imbalance Learning

Joint Sparse Learning for Classification Ensemble.

Comparison among Methods of Ensemble Learning

Ensemble Transfer Learning Algorithm

Ensemble Methods: Foundations and Algorithms

Exploration of Classification Confidence in Ensemble Learning.

A New Rotation Forest Ensemble Algorithm

A Double Penalty Model for Ensemble Learning

Classifiers Combination with Observational Learning

Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [review Article]

Ensemble Method for Classification of High-Dimensional Data.

Ensemble learning: A survey

Clusterer ensemble

Less Is More: A Comprehensive Framework for the Number of Components of Ensemble Classifiers

Ensemble Learning Method for Classification: Integrating Data Envelopment Analysis with Machine Learning

A Comparison of Ensemble Methods in Financial Market Prediction.

Classifier Subset Selection Based on Classifier Representation and Clustering Ensemble

Cocktail Ensemble for Regression

A Probabilistic Ensemble Pruning Algorithm