Abstract:We propose the "study strap ensemble", which combines advantages of two common approaches to fitting prediction models when multiple training datasets ("studies") are available: pooling studies and fitting one model versus averaging predictions from multiple models each fit to individual studies. The study strap ensemble fits models to bootstrapped datasets, or "pseudo-studies." These are generated by resampling from multiple studies with a hierarchical resampling scheme that generalizes the randomized cluster bootstrap. The study strap is controlled by a tuning parameter that determines the proportion of observations to draw from each study. When the parameter is set to its lowest value, each pseudo-study is resampled from only a single study. When it is high, the study strap ignores the multi-study structure and generates pseudo-studies by merging the datasets and drawing observations like a standard bootstrap. We empirically show the optimal tuning value often lies in between, and prove that special cases of the study strap draw the merged dataset and the set of original studies as pseudo-studies. We extend the study strap approach with an ensemble weighting scheme that utilizes information in the distribution of the covariates of the test dataset. Our work is motivated by neuroscience experiments using real-time neurochemical sensing during awake behavior in humans. Current techniques to perform this kind of research require measurements from an electrode placed in the brain during awake neurosurgery and rely on prediction models to estimate neurotransmitter concentrations from the electrical measurements recorded by the electrode. These models are trained by combining multiple datasets that are collected in vitro under heterogeneous conditions in order to promote accuracy of the models when applied to data collected in the brain. A prevailing challenge is deciding how to combine studies or ensemble models trained on different studies to enhance model generalizability. Our methods produce marked improvements in simulations and in this application. All methods are available in the studyStrap CRAN package.

Cross-validation Approaches for Multi-study Predictions

Bayesian nonparametric cross-study validation of prediction methods

Optimal ensemble construction for multistudy prediction with applications to mortality estimation

Heart Failure prediction on diversified datasets to improve generalizability using 2-Level Stacking

Simultaneous prediction of multiple outcomes using revised stacking algorithms

Empirical investigation of multi-source cross-validation in clinical machine learning

Hierarchical resampling for bagging in multistudy prediction with applications to human neurochemical sensing

Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure

Diagnosis of Inflammatory Bowel Disease and Colorectal Cancer through Multi-View Stacked Generalization Applied on Gut Microbiome Data

Privacy-Preserving Stacking with Application to Cross-organizational Diabetes Prediction.

Empirical investigation of multi-source cross-validation in clinical ECG classification

Predictive Classifier for Cardiovascular Disease Based on Stacking Model Fusion

Using cross-validation to evaluate predictive accuracy of survival risk classifiers based on high-dimensional data

Sequential double cross-validation for assessment of added predictive ability in high-dimensional omic applications

Error Reduction from Stacked Regressions

Pitfalls and Remedies for Cross Validation with Multi-trait Genomic Prediction Methods

Investigation of model stacking for drug sensitivity prediction

Multi-Model Stacking Ensemble Learning for Dropout Prediction in MOOCs

Cross-validation in high-dimensional spaces: a lifeline for least-squares models and multi-class LDA

Cross-validation: what does it estimate and how well does it do it?