Abstract:Importance Despite the potential of machine learning to improve multiple aspects of patient care, barriers to clinical adoption remain. Randomized clinical trials (RCTs) are often a prerequisite to large-scale clinical adoption of an intervention, and important questions remain regarding how machine learning interventions are being incorporated into clinical trials in health care. Objective To systematically examine the design, reporting standards, risk of bias, and inclusivity of RCTs for medical machine learning interventions. Evidence Review In this systematic review, the Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science Core Collection online databases were searched and citation chasing was done to find relevant articles published from the inception of each database to October 15, 2021. Search terms for machine learning, clinical decision-making, and RCTs were used. Exclusion criteria included implementation of a non-RCT design, absence of original data, and evaluation of nonclinical interventions. Data were extracted from published articles. Trial characteristics, including primary intervention, demographics, adherence to the CONSORT-AI reporting guideline, and Cochrane risk of bias were analyzed. Findings Literature search yielded 19 737 articles, of which 41 RCTs involved a median of 294 participants (range, 17-2488 participants). A total of 16 RCTS (39%) were published in 2021, 21 (51%) were conducted at single sites, and 15 (37%) involved endoscopy. No trials adhered to all CONSORT-AI standards. Common reasons for nonadherence were not assessing poor-quality or unavailable input data (38 trials [93%]), not analyzing performance errors (38 [93%]), and not including a statement regarding code or algorithm availability (37 [90%]). Overall risk of bias was high in 7 trials (17%). Of 11 trials (27%) that reported race and ethnicity data, the median proportion of participants from underrepresented minority groups was 21% (range, 0%-51%). Conclusions and Relevance This systematic review found that despite the large number of medical machine learning–based algorithms in development, few RCTs for these technologies have been conducted. Among published RCTs, there was high variability in adherence to reporting standards and risk of bias and a lack of participants from underrepresented minority groups. These findings merit attention and should be considered in future RCT design and reporting.

Statistical Thinking, Machine Learning

A Guide for the Application of Statistics in Biomedical Studies Concerning Machine Learning and Artificial Intelligence

Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review

An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction

Randomized Clinical Trials of Machine Learning Interventions in Health Care

Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning

Machine Learning for Benchmarking Critical Care Outcomes

Biostatistiek versus machine learning: van traditionele predictiemodellen naar geautomatiseerde medische analyse

Evaluating Machine Learning Models against Clinical Protocols for Enhanced Interpretability and Continuity of Care

A clinician's guide to understanding and critically appraising machine learning studies: a checklist for Ruling Out Bias Using Standard Tools in Machine Learning (ROBUST-ML)

Clinical Applications of Machine Learning

Integrating Machine Learning into Statistical Methods in Disease Risk Prediction Modeling: A Systematic Review

Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness

A critical moment in machine learning in medicine: on reproducible and interpretable learning

Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data

Statistical and machine learning methods for cancer research and clinical practice: A systematic review

Clinical performance of automated machine learning: A systematic review

Predicting Outcome in Patients with Brain Injury: Differences between Machine Learning versus Conventional Statistics

A review of model evaluation metrics for machine learning in genetics and genomics

Statistical hypothesis testing versus machine-learning binary classification: distinctions and guidelines