Abstract:BACKGROUND AND PURPOSE:Artificial intelligence models in radiology are frequently developed and validated using data sets from a single institution and are rarely tested on independent, external data sets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multicenter artificial intelligence competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND METHODS:In total, 1201 anonymized, full-head NCCT clinical scans from 5 institutions were pooled to form the data set. The data set encompassed studies with normal findings as well as those with pathologies, including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these, task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed the performance of each model. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each artificial intelligence model, along with the area under the ROC curve. RESULTS:Four teams processed 1177 studies. The median age of patients was 62 years, with an interquartile range of 33 years. Nineteen teams from various academic institutions registered for the competition. Of these, 4 teams submitted their final results. No commercial entities participated in the competition. For task 1, areas under the ROC curve ranged from 0.49 to 0.59. For task 2, two teams completed the task with area under the ROC curve values of 0.57 and 0.52. For task 3, teams had little-to-no agreement with the ground truth. CONCLUSIONS:To assess the performance of artificial intelligence models in real-world clinical scenarios, we analyzed their performance in the ASFNR Artificial Intelligence Competition. The first ASFNR Competition underscored the gap between expectation and reality; and the models largely fell short in their assessments. As the integration of artificial intelligence tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.

How can we do better ? Pitfalls in biomedical challenge design and how to address them

Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions.

How to Exploit Weaknesses in Biomedical Challenge Design and Organization.

BIAS: Transparent reporting of biomedical image analysis challenges

Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems

How can we learn (more) from challenges? A statistical approach to driving future algorithm development

Why is the Winner the Best?

Beyond rankings: Learning (more) from algorithm validation

A Framework for Challenge Design: Insight and Deployment Challenges to Address Medical Image Analysis Problems

Why Rankings of Biomedical Image Analysis Competitions Should Be Interpreted with Care

Understanding metric-related pitfalls in image analysis validation

Challenge Results Are Not Reproducible

The Medical Segmentation Decathlon

Community challenges in biomedical text mining over 10 years: success, failure and the future

AI Competitions and Benchmarks: The life cycle of challenges and benchmarks

IDRiD: Diabetic Retinopathy - Segmentation and Grading Challenge.

Abstract: MOOD 2020

Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition

Metrics reloaded: Recommendations for image analysis validation

Ten years of image analysis and machine learning competitions in dementia