Abstract:BACKGROUND AND PURPOSE:Artificial intelligence models in radiology are frequently developed and validated using data sets from a single institution and are rarely tested on independent, external data sets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multicenter artificial intelligence competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND METHODS:In total, 1201 anonymized, full-head NCCT clinical scans from 5 institutions were pooled to form the data set. The data set encompassed studies with normal findings as well as those with pathologies, including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these, task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed the performance of each model. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each artificial intelligence model, along with the area under the ROC curve. RESULTS:Four teams processed 1177 studies. The median age of patients was 62 years, with an interquartile range of 33 years. Nineteen teams from various academic institutions registered for the competition. Of these, 4 teams submitted their final results. No commercial entities participated in the competition. For task 1, areas under the ROC curve ranged from 0.49 to 0.59. For task 2, two teams completed the task with area under the ROC curve values of 0.57 and 0.52. For task 3, teams had little-to-no agreement with the ground truth. CONCLUSIONS:To assess the performance of artificial intelligence models in real-world clinical scenarios, we analyzed their performance in the ASFNR Artificial Intelligence Competition. The first ASFNR Competition underscored the gap between expectation and reality; and the models largely fell short in their assessments. As the integration of artificial intelligence tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.

A reproducible evaluation of ANTs similarity metric performance in brain image registration

An Image Registration Approach Based on 3D Geometric Projection Similarity of the Human Head

NRAAF: A Framework for Comparative Analysis of fMRI Registration Algorithms and Their Impact on Resting-State Neuroimaging Accuracy

Registration of IRT and visible light images in neurosurgery: analysis and comparison of automatic intensity-based registration approaches

Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain

Block-Based Statistics for Robust Non-parametric Morphometry.

Accelerating Neuroimage Registration Through Parallel Computation of Similarity Metric

The ANTsX Ecosystem for Mapping the Mouse Brain

Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition

Digital Analysis of Smart Registration Methods for Magnetic Resonance Images in Public Healthcare

A Reproducible Comparative Study on Registration Packages for Toddler MRI Images

Evaluating Similarity Measures for Brain Image Registration

Deep learning-based affine medical image registration for multimodal minimal-invasive image-guided interventions – A comparative study on generalizability

Automatic 3D Intersubject Registration of MR Volumetric Data in Standardized Talairach Space

Impact of Initialization on Intra-subject Pediatric Brain MR Image Registration: A Comparative Analysis between SyN ANTs and Deep Learning-Based Approaches

Evaluation of Five Diffeomorphic Image Registration Algorithms for Mouse Brain Magnetic Resonance Microscopy.

RegQCNET: Deep quality control for image-to-template brain MRI affine registration

A comparative study of rigid-body registration algorithms for the alignment of longitudinal structural MRI of the brain

Differentiable Optimization of Similarity Scores Between Models and Brains

FireANTs: Adaptive Riemannian Optimization for Multi-Scale Diffeomorphic Registration