Abstract:Ensemble methods use multiple classifiers to achieve better decisions than could be achieved using any of the constituent classifiers alone. However, both theoretical and experimental evidence have shown that very large ensembles are not necessarily superior, and small ensembles can often achieve better results. In this paper, we show how to combine a set of weak classifiers into a robust ensemble by using a joint sparse representation method, which assigns a sparse coefficient vector to the decision of each classifier. The sparse vector contains many zero entries, and thus the final ensemble only employs a small number of classifiers, corresponding to non-zero entries. Training data are partitioned into several sub-groups to generate sub-underdetermined systems. The joint sparse method enables these sub-groups to then share their information about individual classifiers, to obtain an improved overall classification. Partitioning the training dataset into subgroups makes the proposed joint sparse ensemble method parallelizable, making it suitable for large scale problems. In contrast, previous work on sparse approaches to ensemble learning was limited to datasets smaller than the number of classifiers. Two different strategies are described for generating the sub-underdetermined systems, and experiments show these to be effective when tested with two different data manipulation methods. Experiments evaluate the performance of the joint sparse ensemble learning method in comparison to five other state-of-the-art methods from the literature, each designed to train small and efficient ensembles. Results suggest that joint sparse ensemble learning outperforms other algorithms on most datasets.

A Probabilistic Ensemble Pruning Algorithm

Predictive Ensemble Pruning by Expectation Propagation

A Novel Bayesian Ensemble Pruning Method

Margin-Based Pareto Ensemble Pruning: an Ensemble Pruning Algorithm That Learns to Search Optimized Ensembles.

Ensemble Pruning Via Constrained Eigen-Optimization

Ensemble Pruning: A Submodular Function Maximization Perspective.

Ensemble Pruning Via Individual Contribution Ordering

Pareto Ensemble Pruning

Ensemble Pruning Based on Objection Maximization with a General Distributed Framework.

A Rough Sets Based Pruning Method for Bagging Ensemble

A pruning algorithm for training neural network ensembles

Ensemble Pruning Via Base-Classifier Replacement

A ranking-based strategy to prune variable selection ensembles.

Spread Binary Artificial Fish Swarm Algorithm Combined with Double-Fault Measure for Ensemble Pruning

Joint Sparse Learning for Classification Ensemble.

A New Metric For Greedy Ensemble Pruning

An optimal pruning algorithm of classifier ensembles: dynamic programming approach

Multilayer Ensemble Pruning Via Novel Multi-sub-swarm Particle Swarm Optimization.

Ensemble Pruning Based on Frequent Patterns

Combining Weak-Link Co-Evolution Binary Artificial Fish Swarm Algorithm and Complementarity Measure for Ensemble Pruning

Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles