Abstract:BACKGROUND AND PURPOSE:Artificial intelligence models in radiology are frequently developed and validated using data sets from a single institution and are rarely tested on independent, external data sets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multicenter artificial intelligence competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND METHODS:In total, 1201 anonymized, full-head NCCT clinical scans from 5 institutions were pooled to form the data set. The data set encompassed studies with normal findings as well as those with pathologies, including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these, task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed the performance of each model. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each artificial intelligence model, along with the area under the ROC curve. RESULTS:Four teams processed 1177 studies. The median age of patients was 62 years, with an interquartile range of 33 years. Nineteen teams from various academic institutions registered for the competition. Of these, 4 teams submitted their final results. No commercial entities participated in the competition. For task 1, areas under the ROC curve ranged from 0.49 to 0.59. For task 2, two teams completed the task with area under the ROC curve values of 0.57 and 0.52. For task 3, teams had little-to-no agreement with the ground truth. CONCLUSIONS:To assess the performance of artificial intelligence models in real-world clinical scenarios, we analyzed their performance in the ASFNR Artificial Intelligence Competition. The first ASFNR Competition underscored the gap between expectation and reality; and the models largely fell short in their assessments. As the integration of artificial intelligence tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.

224 Artificial Intelligence-based Decision Support Predicts Requirement for Neurosurgical Intervention in Acute Traumatic Brain Injury: Automated Surgical Intervention Support Tool (ASIST-TBI) Development, Validation and Simulated Prospective Deployment

Validation of a Deep Learning Model for Traumatic Brain Injury Detection and NIRIS Grading on Non-Contrast CT: a Multi-Reader Study with Promising Results and Opportunities for Improvement.

Using an artificial neural network to predict traumatic brain injury

Validation of the NeuroImaging Radiological Interpretation System for Acute Traumatic Brain Injury

Development of a Multimodal Machine Learning-Based Prognostication Model for Traumatic Brain Injury Using Clinical Data and Computed Tomography Scans: A CENTER-TBI and CINTER-TBI Study

Enhancing Hospital Course and Outcome Prediction in Patients with Traumatic Brain Injury: A Machine Learning Study.

Clinical Impact of an AI Decision Support System for Detection of Intracranial Hemorrhage in CT Scans

Machine Learning Based Prediction of Imminent ICP Insults During Neurocritical Care of Traumatic Brain Injury

AI-Based Decision Support System for Traumatic Brain Injury: A Survey

A Computer-Assisted System for Early Mortality Risk Prediction in Patients with Traumatic Brain Injury Using Artificial Intelligence Algorithms in Emergency Room Triage

Predictors of 30-Day Mortality and Morbidity Following Craniotomy for Traumatic Brain Injury: An ACS NSQIP Database Analysis

Automated Detection of Intracranial Hemorrhage from Head CT Scans Applying Deep Learning Techniques in Traumatic Brain Injuries: A Comparative Review

Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition

Machine learning‐based models to predict the need for neurosurgical intervention after moderate traumatic brain injury

Potential Applications and Ethical Considerations for Artificial Intelligence in Traumatic Brain Injury Management

Machine Learning Algorithm Predicts Mortality Risk in Intensive Care Unit for Patients with Traumatic Brain Injury

Automatic lesion analysis for increased efficiency in outcome prediction of traumatic brain injury

Creating and validating a neurosurgical intervention rule-out tool for patients with mild traumatic brain injury and isolated subdural hematoma: a 5-year, six-center retrospective cohort study

Use of artificial intelligence to support prehospital traumatic injury care: A scoping review

Three dimensional convolutional neural network-based automated detection of midline shift in traumatic brain injury cases from head computed tomography scans

Development and validation of a prehospital-stage prediction tool for traumatic brain injury: a multicentre retrospective cohort study in Korea