Abstract:Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.

An Ensemble Model For Error Modeling With Pseudoinverse Learning Algorithm

Genetic Ensemble of Extreme Learning Machine

Learning with Pseudo-Ensembles

A Double Penalty Model for Ensemble Learning

Deep Ensembles: A Loss Landscape Perspective

Ensemble with Estimation: Seeking for Optimization in Class Noisy Data

Robust Modeling of Unknown Dynamical Systems via Ensemble Averaged Learning

Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function

Autoencoder, Low Rank Approximation and Pseudoinverse Learning Algorithm.

LEARNING MULTIPLE CAUSES BY COMPETITION ENHANCED LEAST MEAN SQUARE ERROR RECONSTRUCTION

Joint Training of Deep Ensembles Fails Due to Learner Collusion

An Evolution Strategy Assisted by an Ensemble of Local Gaussian Process Models.

Ensemble Adversarial Defense via Integration of Multiple Dispersed Low Curvature Models

An Ensemble Model for Combating Label Noise

On Deep Ensemble Learning from a Function Approximation Perspective

Ex uno plures: Splitting One Model into an Ensemble of Subnetworks

Improving Deep Neural Network Ensembles Using Reconstruction Error

EEE, Remediating the failure of machine learning models via a network-based optimization patch

Nudging, Ensemble, and Nudging Ensembles for Data Assimilation in the Presence of Model Error

On Optimizing Ensemble Models using Column Generation

Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling