Abstract:Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.

Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles

Ex uno plures: Splitting One Model into an Ensemble of Subnetworks

Deep Bayesian Active Learning with Image Data

Deep Ensembles: A Loss Landscape Perspective

Subdomain contraction in deep networks for robust representation learning

Large-Scale Visual Active Learning with Deep Probabilistic Ensembles

Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study

Bayesian Deep Neural Networks for Supervised Learning of Single-View Depth

An Intelligent Multi-View Active Learning Method Based on a Double-Branch Network

Dynamic Post-Hoc Neural Ensemblers

Deep Bayesian Active Learning, A Brief Survey on Recent Advances

Uncertainty estimation in Deep Learning for Panoptic segmentation

Class-Imbalanced Deep Learning via a Class-Balanced Ensemble

Bayesian DivideMix++ for enhanced learning with noisy labels

Mask-Guided Divergence Loss Improves the Generalization and Robustness of Deep Neural Network

Latent Model Ensemble with Auto-localization

EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

Ensemble Deep Learning Network Model for Dropout Prediction in MOOCs

Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks

Deep Super Learner: A Deep Ensemble for Classification Problems