Abstract:Classic results establish that encouraging predictive diversity improves performance in ensembles of low-capacity models, e.g. through bagging or boosting. Here we demonstrate that these intuitions do not apply to high-capacity neural network ensembles (deep ensembles), and in fact the opposite is often true. In a large scale study of nearly 600 neural network classification ensembles, we examine a variety of interventions that trade off component model performance for predictive diversity. While such interventions can improve the performance of small neural network ensembles (in line with standard intuitions), they harm the performance of the large neural network ensembles most often used in practice. Surprisingly, we also find that discouraging predictive diversity is often benign in large-network ensembles, fully inverting standard intuitions. Even when diversity-promoting interventions do not sacrifice component model performance (e.g. using heterogeneous architectures and training paradigms), we observe an opportunity cost associated with pursuing increased predictive diversity. Examining over 1000 ensembles, we observe that the performance benefits of diverse architectures/training procedures are easily dwarfed by the benefits of simply using higher-capacity models, despite the fact that such higher capacity models often yield significantly less predictive diversity. Overall, our findings demonstrate that standard intuitions around predictive diversity, originally developed for low-capacity ensembles, do not directly apply to modern high-capacity deep ensembles. This work clarifies fundamental challenges to the goal of improving deep ensembles by making them more diverse, while suggesting an alternative path: simply forming ensembles from ever more powerful (and less diverse) component models.

Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and Robustness

Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity

Efficient Diversity-Driven Ensemble for Deep Neural Networks

Improving Adversarial Robustness via Promoting Ensemble Diversity.

Improving Adversarial Robustness Via Promoting Ensemble Diversity

DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation

“In-Network Ensemble”: Deep Ensemble Learning with Diversified Knowledge Distillation

An Approach to Improving Diversity of Neural Network Ensemble

Pathologies of Predictive Diversity in Deep Ensembles

Inverse Adversarial Diversity Learning for Network Ensemble

Diversity supporting robustness: Enhancing adversarial robustness via differentiated ensemble predictions

Negatively correlated ensemble against transfer adversarial attacks

The Diversified Ensemble Neural Network.

Joint Training of Deep Ensembles Fails Due to Learner Collusion

Deep Ensembles Work, But Are They Necessary?

A DNN-Ensemble Method for Error Reduction and Training Data Selection in DNN Based Modeling

Exploring Model Learning Heterogeneity for Boosting Ensemble Robustness

Dynamic ensemble selection based on Deep Neural Network Uncertainty Estimation for Adversarial Robustness

Deep Ensembles: A Loss Landscape Perspective

Ensembles of Artificial Example Based Neural Networks.

The Disparate Benefits of Deep Ensembles