Abstract:Ensemble methods use multiple classifiers to achieve better decisions than could be achieved using any of the constituent classifiers alone. However, both theoretical and experimental evidence have shown that very large ensembles are not necessarily superior, and small ensembles can often achieve better results. In this paper, we show how to combine a set of weak classifiers into a robust ensemble by using a joint sparse representation method, which assigns a sparse coefficient vector to the decision of each classifier. The sparse vector contains many zero entries, and thus the final ensemble only employs a small number of classifiers, corresponding to non-zero entries. Training data are partitioned into several sub-groups to generate sub-underdetermined systems. The joint sparse method enables these sub-groups to then share their information about individual classifiers, to obtain an improved overall classification. Partitioning the training dataset into subgroups makes the proposed joint sparse ensemble method parallelizable, making it suitable for large scale problems. In contrast, previous work on sparse approaches to ensemble learning was limited to datasets smaller than the number of classifiers. Two different strategies are described for generating the sub-underdetermined systems, and experiments show these to be effective when tested with two different data manipulation methods. Experiments evaluate the performance of the joint sparse ensemble learning method in comparison to five other state-of-the-art methods from the literature, each designed to train small and efficient ensembles. Results suggest that joint sparse ensemble learning outperforms other algorithms on most datasets.

Predictive Ensemble Pruning by Expectation Propagation

A Probabilistic Ensemble Pruning Algorithm

Ensemble Pruning Via Constrained Eigen-Optimization

Pareto Ensemble Pruning

A Novel Bayesian Ensemble Pruning Method

Margin-Based Pareto Ensemble Pruning: an Ensemble Pruning Algorithm That Learns to Search Optimized Ensembles.

Ensemble Pruning Via Base-Classifier Replacement

Ensemble Pruning Via Individual Contribution Ordering

Ensemble Pruning Based on Objection Maximization with a General Distributed Framework.

Regularization Based Ordering for Ensemble Pruning

Ensemble Pruning: A Submodular Function Maximization Perspective.

Margin distribution and structural diversity guided ensemble pruning

Hierachical Bagging Ensemble Pruning Based on Deleting Worst Base Learner

A pruning algorithm for training neural network ensembles

A ranking-based strategy to prune variable selection ensembles.

Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles

A New Metric For Greedy Ensemble Pruning

The MBPEP: a Deep Ensemble Pruning Algorithm Providing High Quality Uncertainty Prediction

Pruning Variable Selection Ensembles

Joint Sparse Learning for Classification Ensemble.

Developing parsimonious ensembles using ensemble diversity within a reinforcement learning framework