Abstract:Background: AI-based clinical decision support system (CDSS) has important prospects in overcoming the current informational challenges that cancer diseases faced, promoting the homogeneous development of standardized treatment among different geographical regions, and reforming the medical model. However, there are still a lack of relevant indicators to comprehensively assess its decision-making quality and clinical impact, which greatly limits the development of its clinical research and clinical application. This study aims to develop and application an assessment system that can comprehensively assess the decision-making quality and clinical impacts of physicians and CDSS. Methods: Enrolled adjuvant treatment decision stage early breast cancer cases were randomly assigned to different decision-making physician panels (each panel consisted of three different seniority physicians in different grades hospitals), each physician made an independent "Initial Decision" and then reviewed the CDSS report online and made a "Final Decision". In addition, the CDSS and guideline expert groups independently review all cases and generate "CDSS Recommendations" and "Guideline Recommendations" respectively. Based on the design framework, a multi-level multi-indicator system including "Decision Concordance", "Calibrated Concordance", " Decision Concordance with High-level Physician", "Consensus Rate", "Decision Stability", "Guideline Conformity", and "Calibrated Conformity" were constructed. Results: 531 cases containing 2124 decision points were enrolled; 27 different seniority physicians from 10 different grades hospitals have generated 6372 decision opinions before and after referring to the "CDSS Recommendations" report respectively. Overall, the calibrated decision concordance was significantly higher for CDSS and provincial-senior physicians (80.9%) than other physicians. At the same time, CDSS has a higher " decision concordance with high-level physician" (76.3%-91.5%) than all physicians. The CDSS had significantly higher guideline conformity than all decision-making physicians and less internal variation, with an overall guideline conformity variance of 17.5% (97.5% vs. 80.0%), a standard deviation variance of 6.6% (1.3% vs. 7.9%), and a mean difference variance of 7.8% (1.5% vs. 9.3%). In addition, provincial-middle seniority physicians had the highest decision stability (54.5%). The overall consensus rate among physicians was 64.2%. Conclusions: There are significant internal variation in the standardization treatment level of different seniority physicians in different geographical regions in the adjuvant treatment of early breast cancer. CDSS has a higher standardization treatment level than all physicians and has the potential to provide immediate decision support to physicians and have a positive impact on standardizing physicians' treatment behaviors.

Quality control system of Watson for oncology: Artificial intelligence for supporting clinical decisions in oncology.

Artificial Intelligence in Decision-Making for Colorectal Cancer Treatment Strategy: an Observational Study of Implementing Watson for Oncology in a 250-Case Cohort

Concordance between treatment recommendations provided by IBM Watson for Oncology and a multidisciplinary tumor board for breast cancer in China

A meta-analysis of Watson for Oncology in clinical application

Concordance of Treatment Recommendations for Metastatic Non-Small-Cell Lung Cancer Between Watson for Oncology System and Medical Team

Artificial intelligence in urologic oncology: the actual clinical practice results of IBM Watson for Oncology in South Korea

Deployment and Integration of a Cognitive Technology in China: Experiences and Lessons Learned.

Real world study for the concordance between IBM Watson for Oncology and clinical practice in advanced non‐small cell lung cancer patients at a lung cancer center in China

Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board

Concordance Evaluation of an Artificial Intelligence Technology with a Multidisciplinary Tumor Board in Gastric Cancer.

Artificial intelligence and lung cancer treatment decision: agreement with recommendation of multidisciplinary tumor board

Artificial intelligence in clinical decision support systems for oncology

Concordance Between Watson for Oncology and a Multidisciplinary Clinical Decision-Making Team for Gastric Cancer and the Prognostic Implications: Retrospective Study

Comparison of case-based learning using Watson for oncology and traditional method in teaching undergraduate medical students

Precision Oncology and AI: the ACMA System for Personalized Treatment Matching Using Real-World Data.

Can AI Technology Augment Tumor Board Treatment Decisions for Stage II Colon Cancer Care?

Assessing the decision quality of artificial intelligence and oncologists of different experience in different regions in breast cancer treatment

A quantitative quality control method of big data in cancer patients using artificial neural network

Supervision System of AI-based Software As a Medical Device

The Chinese Thoracic Oncology Group (CTONG) Therapeutic Option Scoring System: a Multiple-Parameter Framework to Assess the Value of Lung Cancer Treatment Options

Consistency of CSCO AI with Multidisciplinary Clinical Decision-Making Teams in Breast Cancer: A Retrospective Study