Abstract:Abstract Objective To develop a clinically useful model that estimates the 10 year risk of breast cancer related mortality in women (self-reported female sex) with breast cancer of any stage, comparing results from regression and machine learning approaches. Design Population based cohort study. Setting QResearch primary care database in England, with individual level linkage to the national cancer registry, Hospital Episodes Statistics, and national mortality registers. Participants 141 765 women aged 20 years and older with a diagnosis of invasive breast cancer between 1 January 2000 and 31 December 2020. Main outcome measures Four model building strategies comprising two regression (Cox proportional hazards and competing risks regression) and two machine learning (XGBoost and an artificial neural network) approaches. Internal-external cross validation was used for model evaluation. Random effects meta-analysis that pooled estimates of discrimination and calibration metrics, calibration plots, and decision curve analysis were used to assess model performance, transportability, and clinical utility. Results During a median 4.16 years (interquartile range 1.76-8.26) of follow-up, 21 688 breast cancer related deaths and 11 454 deaths from other causes occurred. Restricting to 10 years maximum follow-up from breast cancer diagnosis, 20 367 breast cancer related deaths occurred during a total of 688 564.81 person years. The crude breast cancer mortality rate was 295.79 per 10 000 person years (95% confidence interval 291.75 to 299.88). Predictors varied for each regression model, but both Cox and competing risks models included age at diagnosis, body mass index, smoking status, route to diagnosis, hormone receptor status, cancer stage, and grade of breast cancer. The Cox model’s random effects meta-analysis pooled estimate for Harrell’s C index was the highest of any model at 0.858 (95% confidence interval 0.853 to 0.864, and 95% prediction interval 0.843 to 0.873). It appeared acceptably calibrated on calibration plots. The competing risks regression model had good discrimination: pooled Harrell’s C index 0.849 (0.839 to 0.859, and 0.821 to 0.876, and evidence of systematic miscalibration on summary metrics was lacking. The machine learning models had acceptable discrimination overall (Harrell’s C index: XGBoost 0.821 (0.813 to 0.828, and 0.805 to 0.837); neural network 0.847 (0.835 to 0.858, and 0.816 to 0.878)), but had more complex patterns of miscalibration and more variable regional and stage specific performance. Decision curve analysis suggested that the Cox and competing risks regression models tested may have higher clinical utility than the two machine learning approaches. Conclusion In women with breast cancer of any stage, using the predictors available in this dataset, regression based methods had better and more consistent performance compared with machine learning approaches and may be worthy of further evaluation for potential clinical use, such as for stratified follow-up.

Overview of model validation for survival regression model with competing risks using melanoma study data

Using cross-validation to evaluate predictive accuracy of survival risk classifiers based on high-dimensional data

Clinical Prediction Models: Model Validation

Nomogram for survival analysis in the presence of competing risks.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Unbiased Prediction and Feature Selection in High-Dimensional Survival Regression

External validation of a Cox prognostic model: principles and methods

A review on statistical and machine learning competing risks methods

Parametric Regression Model for Survival Data: Weibull Regression Model As an Example

Variable selection for competing risk regression models: recommendations for analyzing data from epidemiological studies

Minimum sample size calculations for external validation of a clinical prediction model with a time‐to‐event outcome

Neural Fine-Gray: Monotonic neural networks for competing risks

Competitive Risk Model for Specific Mortality Prediction in Patients with Bladder Cancer: A Population-Based Cohort Study with Machine Learning

Cross-validatory Z-Residual for Diagnosing Shared Frailty Models

Statistical Development and Validation of Clinical Prediction Models

Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study

Confidence intervals for the Cox model test error from cross-validation

Evaluating discriminatory accuracy of models using partial risk-scores in two-phase studies

Evaluating Discrimination of a Lung Cancer Risk Prediction Model Using Partial Risk-Score in a Two-Phase Study

A tutorial on evaluating time-varying discrimination accuracy for survival models used in dynamic decision-making

Development and validation of a competitive risk model in patients with rectal cancer: based on SEER database