Abstract:Background: We propose a decision-referral approach for integrating artificial intelligence (AI) into the breast-cancer screening pathway, whereby the algorithm makes predictions on the basis of its quantification of uncertainty. Algorithmic assessments with high certainty are done automatically, whereas assessments with lower certainty are referred to the radiologist. This two-part AI system can triage normal mammography exams and provide post-hoc cancer detection to maintain a high degree of sensitivity. This study aimed to evaluate the performance of this AI system on sensitivity and specificity when used either as a standalone system or within a decision-referral approach, compared with the original radiologist decision. Methods: We used a retrospective dataset consisting of 1 193 197 full-field, digital mammography studies carried out between Jan 1, 2007, and Dec 31, 2020, from eight screening sites participating in the German national breast-cancer screening programme. We derived an internal-test dataset from six screening sites (1670 screen-detected cancers and 19 997 normal mammography exams), and an external-test dataset of breast cancer screening exams (2793 screen-detected cancers and 80 058 normal exams) from two additional screening sites to evaluate the performance of an AI algorithm on sensitivity and specificity when used either as a standalone system or within a decision-referral approach, compared with the original individual radiologist decision at the point-of-screen reading ahead of the consensus conference. Different configurations of the AI algorithm were evaluated. To account for the enrichment of the datasets caused by oversampling cancer cases, weights were applied to reflect the actual distribution of study types in the screening programme. Triaging performance was evaluated as the rate of exams correctly identified as normal. Sensitivity across clinically relevant subgroups, screening sites, and device manufacturers was compared between standalone AI, the radiologist, and decision referral. We present receiver operating characteristic (ROC) curves and area under the ROC (AUROC) to evaluate AI-system performance over its entire operating range. Comparison with radiologists and subgroup analysis was based on sensitivity and specificity at clinically relevant configurations. Findings: The exemplary configuration of the AI system in standalone mode achieved a sensitivity of 84·2% (95% CI 82·4-85·8) and a specificity of 89·5% (89·0-89·9) on internal-test data, and a sensitivity of 84·6% (83·3-85·9) and a specificity of 91·3% (91·1-91·5) on external-test data, but was less accurate than the average unaided radiologist. By contrast, the simulated decision-referral approach significantly improved upon radiologist sensitivity by 2·6 percentage points and specificity by 1·0 percentage points, corresponding to a triaging performance at 63·0% on the external dataset; the AUROC was 0·982 (95% CI 0·978-0·986) on the subset of studies assessed by AI, surpassing radiologist performance. The decision-referral approach also yielded significant increases in sensitivity for a number of clinically relevant subgroups, including subgroups of small lesion sizes and invasive carcinomas. Sensitivity of the decision-referral approach was consistent across the eight included screening sites and three device manufacturers. Interpretation: The decision-referral approach leverages the strengths of both the radiologist and AI, demonstrating improvements in sensitivity and specificity surpassing that of the individual radiologist and of the standalone AI system. This approach has the potential to improve the screening accuracy of radiologists, is adaptive to the requirements of screening, and could allow for the reduction of workload ahead of the consensus conference, without discarding the generalised knowledge of radiologists. Funding: Vara.

Three scans are better than two for follow-up: An automatic method for finding missed and misidentified lesions in cross-sectional follow-up of oncology patients

A graph-theoretic approach for the analysis of lesion changes and lesions detection review in longitudinal oncological imaging

Performance of an automated registration-based method for longitudinal lesion matching and comparison to inter-reader variability

Reliability of Automated RECIST 1.1 and Volumetric RECIST Target Lesion Response Evaluation in Follow-Up CT—A Multi-Center, Multi-Observer Reading Study

Improving assessment of lesions in longitudinal CT scans: a bi-institutional reader study on an AI-assisted registration and volumetric segmentation workflow

SU‐E‐T‐40: Exploring the Reproducibility of Tumor Volumes Measured by Radiologist, Computer‐Aided Radiologist and Computer Alone

Image registration based automated lesion correspondence pipeline for longitudinal CT data

Scanner-specific optimisation of automated lesion segmentation in MS

An automated methodology for whole-body, multimodality tracking of individual cancer lesions

Two is better than one: longitudinal detection and volumetric evaluation of brain metastases after Stereotactic Radiosurgery with a deep learning pipeline

AI for interpreting screening mammograms: implications for missed cancer in double reading practices and challenging-to-locate lesions

Dynamic AI-assisted Ipsilateral Tissue Matching for Digital Breast Tomosynthesis

Methods for improving colorectal cancer annotation efficiency for artificial intelligence-observer training

Assessing the performance of AI-assisted technicians in liver segmentation, Couinaud division, and lesion detection: a pilot study

Abstract 2595: Accelerated and precise tumor segmentation in NSCLC: A comparative analysis of automated ClickSeg and manual annotation for radiomics

Proof‐of‐concept study of artificial intelligence‐assisted review of CBCT image guidance

Nimg-63. Leveraging Llms For Accurate Differentiation Of Radiation Necrosis And Tumor Progression In Brain Mri Reports: A Study On Automated Scoring And Clinical Implications

A real-world evaluation of the diagnostic accuracy of radiologists using positive predictive values verified from deep learning and natural language processing chest algorithms deployed retrospectively

Cross-Check QA: A Quality Assurance Workflow to Prevent Missed Diagnoses by Alerting Inadvertent Discordance Between the Radiologist and Artificial Intelligence in the Interpretation of High-Acuity CT Scans

Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis

Prediction of upgrade to clinically significant prostate cancer in patients under active surveillance: performance of a fully automated AI‐algorithm for lesion detection and classification