Abstract:Deep neural network ensembles combine the wisdom of multiple deep neural networks to improve the generalizability and robustness over individual networks. It has gained increasing popularity to study deep ensemble techniques in the deep learning community. Some mission-critical applications utilize a large number of deep neural networks to form deep ensembles to achieve desired accuracy and resilience, which introduces high time and space costs for ensemble execution. However, it still remains a critical challenge whether a small subset of the entire deep ensemble can achieve the same or better generalizability and how to effectively identify these small deep ensembles for improving the space and time efficiency of ensemble execution. This paper presents a novel deep ensemble pruning approach, which can efficiently identify smaller deep ensembles and provide higher ensemble accuracy than the entire deep ensemble of a large number of member networks. Our hierarchical ensemble pruning approach (HQ) leverages three novel ensemble pruning techniques. First, we show that the focal diversity metrics can accurately capture the complementary capacity of the member networks of an ensemble, which can guide ensemble pruning. Second, we design a focal diversity based hierarchical pruning approach, which will iteratively find high quality deep ensembles with low cost and high accuracy. Third, we develop a focal diversity consensus method to integrate multiple focal diversity metrics to refine ensemble pruning results, where smaller deep ensembles can be effectively identified to offer high accuracy, high robustness and high efficiency. Evaluated using popular benchmark datasets, we demonstrate that the proposed hierarchical ensemble pruning approach can effectively identify high quality deep ensembles with better generalizability while being more time and space efficient in ensemble decision making.

Improving deep forest by ensemble pruning based on feature vectorization and quantum walks

WCDForest: a Weighted Cascade Deep Forest Model Toward the Classification Tasks

Tri-objective Optimization-Based Cascade Ensemble Pruning for Deep Forest

A New Rotation Forest Ensemble Algorithm

A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

Spread Binary Artificial Fish Swarm Algorithm Combined with Double-Fault Measure for Ensemble Pruning

ForestPrune: Compact Depth-Controlled Tree Ensembles

Hierarchical Pruning of Deep Ensembles with Focal Diversity

A Quantum-Inspired Ensemble Method and Quantum-Inspired Forest Regressors

Improved Discrete Artificial Fish Swarm Algorithm Combined with Margin Distance Minimization for Ensemble Pruning

QC-Forest: a Classical-Quantum Algorithm to Provably Speedup Retraining of Random Forest

Combining Weak-Link Co-Evolution Binary Artificial Fish Swarm Algorithm and Complementarity Measure for Ensemble Pruning

Improving Deep Forest by Confidence Screening.

Growing Deep Forests Efficiently with Soft Routing and Learned Connectivity

Improving Deep Forest by Exploiting High-order Interactions

Deep Forest: Towards an Alternative to Deep Neural Networks.

Deep Forest: Towards an Alternative to Deep Neural Networks

Improve Deep Forest with Learnable Layerwise Augmentation Policy Schedules.

Multilayer Ensemble Pruning Via Novel Multi-sub-swarm Particle Swarm Optimization.

Improve Deep Forest with Learnable Layerwise Augmentation Policy Schedule

On Extreme Pruning of Random Forest Ensembles for Real-time Predictive Applications