Abstract:Uncertainty estimation is crucial for machine learning models to detect out-of-distribution (OOD) inputs. However, the conventional discriminative deep learning classifiers produce uncalibrated closed-set predictions for OOD data. A more robust classifiers with the uncertainty estimation typically require a potentially unavailable OOD dataset for outlier exposure training, or a considerable amount of additional memory and compute to build ensemble models. In this work, we improve on uncertainty estimation without extra OOD data or additional inference costs using an alternative Split-Ensemble method. Specifically, we propose a novel subtask-splitting ensemble training objective, where a common multiclass classification task is split into several complementary subtasks. Then, each subtask's training data can be considered as OOD to the other subtasks. Diverse submodels can therefore be trained on each subtask with OOD-aware objectives. The subtask-splitting objective enables us to share low-level features across submodels to avoid parameter and computational overheads. In particular, we build a tree-like Split-Ensemble architecture by performing iterative splitting and pruning from a shared backbone model, where each branch serves as a submodel corresponding to a subtask. This leads to improved accuracy and uncertainty estimation across submodels under a fixed ensemble computation budget. Empirical study with ResNet-18 backbone shows Split-Ensemble, without additional computation cost, improves accuracy over a single model by 0.8%, 1.8%, and 25.5% on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively. OOD detection for the same backbone and in-distribution datasets surpasses a single model baseline by, correspondingly, 2.2%, 8.1%, and 29.6% mean AUROC.

Divergent Ensemble Networks: Enhancing Uncertainty Estimation with Shared Representations and Independent Branching

The Diversified Ensemble Neural Network.

Sequential Bayesian Neural Subnetwork Ensembles

Dynamic ensemble of regression neural networks based on predictive uncertainty

Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles

Deep Ensembles Work, But Are They Necessary?

A DNN-Ensemble Method for Error Reduction and Training Data Selection in DNN Based Modeling

Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity

Deep Ensembles: A Loss Landscape Perspective

Efficient Diversity-Driven Ensemble for Deep Neural Networks

Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and Robustness

Ensemble Neural Networks (ENN): A gradient-free stochastic method

Robust Modeling of Unknown Dynamical Systems via Ensemble Averaged Learning

DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation

Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation

Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation

Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification

DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles

An Approach to Improving Diversity of Neural Network Ensemble

“In-Network Ensemble”: Deep Ensemble Learning with Diversified Knowledge Distillation

Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting