False-Negative and False-Positive Outcomes of an Artificial Intelligence System and Observers on Brain Metastasis Detection: Secondary Analysis of a Prospective, Multicentre, Multireader Study
Xiao Luo,Yadi Yang,Shaohan Yin,Hui Li,Wei-Jing Zhang,Gui-Xiao Xu,Weixiong Fan,Dechun Zheng,Jianpeng Li,Dinggang Shen,Yaozong Gao,Ying Shao,Xaiohua Ban,Jing Li,Shan-Shan Lian,Cheng Zhang,Lidi Ma,Cuiping Lin,Yingwei Luo,Fan Zhou,Shiyuan Wang,Ying Sun,Rong Zhang,Chuanmiao Xie
DOI: https://doi.org/10.2139/ssrn.4071504
2022-01-01
SSRN Electronic Journal
Abstract:Background: Errors have seldom been evaluated in computer-aided detection studies of brain metastases. This study aimed to analyze false negatives (FNs) and false positives (FPs) generated by a brain metastasis detection system (BMDS) and by readers, and to investigate the lesion characteristics associated with FNs. Methods: A deep learning-based BMDS was developed using magnetic resonance imaging and prospectively validated in a multicenter, multireader study. Ad hoc secondary analysis was restricted to the prospective participants (148 with 1,066 brain metastases and 152 normal controls). Three trainees and three experienced radiologists read the same images without and with the BMDS. The number of FNs and FPs per patient, jackknife alternative free-response receiver operating characteristic figure of merit (FOM) values, and lesion features associated with FNs were analyzed for the BMDS and the readers using binary logistic regression. Findings: The FNs, FPs, and the FOM value of the stand-alone BMDS were 0∙49, 0∙39, and 0∙97, respectively. Compared with independent reading, BMDS-assisted reading generated 79% fewer FNs (1∙98 vs. 0∙42, p<0∙001); 41% more FPs (0∙17 vs. 0∙24, p<0∙001) but 125% more FPs for trainees (p<0∙001); and a higher FOM value (0∙87 vs. 0∙98, p<0∙001). Lesions with a small size, greater number, an irregular shape, and lower signal intensity were strongly associated with FNs for readers. However, small, irregular, and necrotic lesions were more frequently found in FNs generated by the BMDS. The FPs mainly resulted from small blood vessels for BMDS and readers. Interpretation: The significant improvement in radiologists’ performance supports the BMDS as a tool for brain metastasis detection. However, attention should be paid to FPs and small lesions with lower enhancement, especially for less-experienced radiologists.Funding: None to declare. Declaration of Interest: We declare that there are no conflicts of interest.Ethical Approval: The main study was approved by the institutional review boards of all participating institutions (No. B2021-198-01), and informed consent was obtained from the patients prospectively.