Abstract:Background: Molecular-biomarker-based prediction models, whether based on statistics or artificial intelligence, are becoming increasingly popular in the medical community as an alternative or complement to more traditional prediction tools, such as staging systems. Population-based cohorts are the preferred study design for building and validating such models, but they are generally expensive and frequently unfeasible, particularly, when the outcome of interest is rare, and when the collected data include expensive or difficult-to-obtain molecular biomarkers. As such, studies often develop or validate risk models in cohorts obtained based on data availability, rather than on the representativeness of a target population. Nested case-control (NCC) is an efficient study design for building and validating such models in a rare outcome setting, but it requires a fully enumerated source population, and appropriate methodologies to accommodate the under-sampling of the controls and any matching. Unfortunately, these methodologies are not systematically described in a form easily accessible to clinicians. Methods: We systematically characterized how to correctly develop and evaluate the performance of prediction models in NCC cohorts, by weighing the subjects based on sampling probabilities to account for the sub-cohort sampling. We illustrated the use of weighted metrics for the NCC, with a validation of a model that predicts breast cancer development - BOADICEA version 5 - in the population-based Rotterdam study. We used the C-index, threshold-based metrics, observed-to-expected events ratio (O/E ratio), calibration slope, and decision curve analysis as performance metrics. We compared the metrics obtained in the full cohort with those obtained in NCC cohorts sampled from the Rotterdam study, with and without a matched design. Results: NCC cohorts of 326 women were derived from the full cohort of 4,377 women from the Rotterdam Study. Performance metrics without weight adjustment were biased on the NCC cohorts: the unweighted C-index of the BOADICEA model was 0.61 (0.58-0.63) for the unmatched design. However, with weight adjustment, the C-index in the NCC cohorts (0.65 (0.61-0.69)) corresponds to the C-index in the full cohort (0.65 (0.62-0.69)), despite the NCC cohort being much smaller in size. Similarly, weighted adjustments of O/E ratio, threshold-based metrics, and net benefit for decision curves were unbiased estimates of the corresponding metrics in the full cohort, while the corresponding unweighted metrics were biased. Conclusions: We provided a practical guide for clinicians on how to develop and validate prediction models in NCC cohorts. We showed that this study design is an efficient solution, in case of expensive or difficult-to-obtain biomarkers, and when the outcome is rare; but the performance metrics must be appropriately adjusted to the sampling procedure. Citation Format: Barbara Rentroia-Pacheco, Domenico Bellomo, Inge MM Lakeman, Marlies Wakkee, Loes M. Hollestein, David van Klaveren. Avoiding the pitfalls in developing and validating prediction models for rare outcomes in nested case-control [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular s); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl) nr 4868.

Robust Clustering With Subpopulation-Specific Deviations

Supervised Robust Profile Clustering

Clusters, determinants, and short- and long-term effects of coexisting multiple high-risk factors in pregnancy on mothers and their children’s health: study protocol

Functional clustering in nested designs: Modeling variability in reproductive epidemiology studies

Clustering Based BMI Indexing for Child Disease Prone-Probability Prediction

Shared and ethnic background site-specific dietary patterns in the Hispanic Community Health Study/Study of Latinos (HCHS/SOL)

Derivation of outcome-dependent dietary patterns for low-income women obtained from survey data using a supervised weighted overfitted latent class analysis

Adverse Subpopulation Regression for Multivariate Outcomes with High-Dimensional Predictors

A Joint Modeling Approach for Clustering Mixed-Type Multivariate Longitudinal Data: Application to the CHILD Cohort Study

Privacy-protecting multivariable-adjusted distributed regression analysis for multi-center pediatric study

A search for factors associated with reduced carbohydrate intake and NTD risk in two population‐based studies

Survival Cluster Analysis

A cluster-based ensemble approach for congenital heart disease prediction

Analyzing Multiple Social Determinants of Health Using Different Clustering Methods

Low-income women's reproductive weight patterns empirically based clusters of prepregnant, gestational, and postpartum weights

Abstract 4868: Avoiding the pitfalls in developing and validating prediction models for rare outcomes in nested case-control

Cohort profile: The Camden Study - a pregnancy cohort study of pregnancy complications and birth outcomes in Camden, New Jersey, USA

Estimating racial health disparities among adverse birth outcomes as deviations from the population rates

Feature Selection Approaches for Newborn Birthweight Prediction in Multiple Linear Regression Models

Machine learning analysis with population data for prepregnancy and perinatal risk factors for the neurodevelopmental delay of offspring

Kernel Machine and Distributed Lag Models for Assessing Windows of Susceptibility to Environmental Mixtures in Children's Health Studies