A Comparison Between CSCO AI and Clinicians in Breast Cancer: A Double-Blind, Controlled Phase III Trial

Jianbin Li,Yang Yuan,Li Bian,Qiang Lin,Hua Yang,Li Ma,Ling Xin,Feng Li,Shaohua Zhang,Tao Wang,Yinhua Liu,Zefei Jiang

DOI: https://doi.org/10.2139/ssrn.4153463

2022-01-01

SSRN Electronic Journal

Abstract:Background: The efficiency of regimens for selection by clinicians in compliance with clinical guidelines is enhanced by digital clinical decision support. We are building a clinical decision support system (CSCO AI) for breast cancer patients to improve the efficiency of clinical decision-making. We aimed to assess cancer treatment regimens given by CSCO AI and different levels of clinicians.Methods: 400 breast cancer patients were screened from the CSCO database, including 100 cases of a new diagnosis, post-operation, before the first-line, and after first-line therapy respectively. These 400 cases were randomly divided into two volumes according to hormone receptor and HER2 status. Clinicians with similar levels were randomly assigned one of the volumes (200 cases). CSCO AI was asked to assess all cases. Two reviewers with more than 10 years of clinical experience were asked to evaluate the regimens from clinicians and CSCO AI independently. Regimens were masked before evaluation. After evaluation, a third reviewer with more than 30 years of clinical experience gave the final scores if there were different points between the prior two reviewers. The primary outcome was the proportion of high-level conformity. The second outcome was a concordance rate, which was defined as the percentage of the regimens with the same level of conformity (points) or the uniform decisions in radiotherapy and target therapy between clinicians and CSCO AI.Findings: A total of 14 clinicians from 4 hospitals with distinct professions and levels participated in this trial, 6 surgeons and 8 physicians. The overall concordance between clinicians and CSCO AI was 73.9% (3621/4900). It was 78.8%(2757/3500) in the early-stage, higher than that in the metastatic stage (61.7% [864/1400], p<0.001). The concordance was 90.7% (635/700) and 89.4% (626/700) in adjuvant radiotherapy and adjuvant target therapy respectively, while the proportion was only 56.4% (395/700) in second-line therapy. High-level conformity in CSCO AI was 95.8% (479/500, 95%CI:94.0%-97.6%), significantly higher than that in clinicians (90.8%, 3178/3500, 95%CI:89.8%-91.8%). Considering professions, the high-level conformity of surgeons was 85.9%, lower than that of CSCO AI (OR=0.25,95%CI: 0.16-0.41). The most noteworthy difference in high-level conformity between surgeons and CSCO AI was in first-line therapy (OR=0.06, 95%CI:0.01-0.41). When clinicians were divided according to their levels, the high-level conformity of clinicians was significantly lower than that of CSCO AI (p<0.05). However, there was no statistical significance between CSCO AI and level 2 or 3 clinicians. Scores from CSCO AI were significantly higher than that from clinicians in each treatment stage, especially in first-line therapy (Δ=0.24,p<0.001). Physicians got 2.75±0.61 scores in second-line therapy, significantly higher than that CSCO AI did (2.60±0.77, p=0.006).Interpretation: There are differences in decision-making when compared with the CSCO AI system and clinicians according to diverse professions and levels. Clinical decision support for breast cancer was superior for most process outcomes except for second-line therapy. Funding: This work was supported by the Capital Clinical Characteristic Application Research(Grant numberZ181100001718215) and the innovative projects of PLA generalhospital (Grant number CX19011)Declaration of Interest: All authors declare no competing interestsEthical Approval: This study was approved by the Henan tumour hospital ethic committee(2018019)