Abstract:Medical applications of machine learning (ML) have experienced a surge in popularity in recent years. The intensive care unit (ICU) is a natural habitat for ML given the abundance of available data from electronic health records. Models have been proposed to address numerous ICU prediction tasks like the early detection of complications. While authors frequently report state-of-the-art performance, it is challenging to verify claims of superiority. Datasets and code are not always published, and cohort definitions, preprocessing pipelines, and training setups are difficult to reproduce. This work introduces Yet Another ICU Benchmark (YAIB), a modular framework that allows researchers to define reproducible and comparable clinical ML experiments; we offer an end-to-end solution from cohort definition to model evaluation. The framework natively supports most open-access ICU datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future ICU datasets. Combined with a transparent preprocessing pipeline and extensible training code for multiple ML and deep learning models, YAIB enables unified model development. Our benchmark comes with five predefined established prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and length of stay) developed in collaboration with clinicians. Adding further tasks is straightforward by design. Using YAIB, we demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance - often more so than model class - indicating an urgent need for YAIB as a holistic benchmarking tool. We provide our work to the clinical ML community to accelerate method development and enable real-world clinical implementations. Software Repository: https://github.com/rvandewater/YAIB.

Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial

Evaluation of machine learning algorithms for health and wellness applications: A tutorial

Empirical investigation of multi-source cross-validation in clinical ECG classification

Overview of model validation for survival regression model with competing risks using melanoma study data

Clinical Prediction Models: Model Validation

A Multidatabase ExTRaction PipEline (METRE) for Facile Cross Validation in Critical Care Research

Practical approaches in evaluating validation and biases of machine learning applied to mobile health studies

Empirical investigation of multi-source cross-validation in clinical machine learning

Is Cross-Validation the Gold Standard to Evaluate Model Performance?

Targeted Development and Validation of Clinical Prediction Models in Secondary Care Settings: Opportunities and Challenges for Electronic Health Record Data

Cross-validation in high-dimensional spaces: a lifeline for least-squares models and multi-class LDA

Using cross-validation to evaluate predictive accuracy of survival risk classifiers based on high-dimensional data

Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML

Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications

External validation of machine learning models - registered models and adaptive sample splitting

Efficient, adaptive cross-validation for tuning and comparing models, with application to drug discovery

Don't Waste Your Time: Early Stopping Cross-Validation

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Cross-Validation for Nonlinear Mixed Effects Models

From Theory to Practice: Implementing and Evaluating e-Fold Cross-Validation