Abstract:Computer-aided detection (CADe), computer-aided diagnosis (CADx), and computer-aided simple triage (CAST), which incorporate artificial intelligence (AI) and machine learning (ML), are continually undergoing post-market improvement. Therefore, understanding the evaluation and approval process of improved products is important. This study intended to conduct a comprehensive survey of AI/ML-based CAD products approved by the U.S. Food and Drug Administration (FDA) that had been improved post-market to gain insights into the efficacy and safety required for market approval. A survey of the product code database published by the FDA identified eight products that were improved post-market. The methods used to evaluate the performance of improvements were analysed, and post-market improvements were approved with retrospective data. Reader study testing (RT) or software standalone testing (SA) procedures were conducted retrospectively. Six RT procedures were conducted because of modifications to the intended use. An average of 17.3 readers (minimum 14, maximum 24) participated, and the area under the curve (AUC) was considered the primary endpoint. The addition of study learning data that did not change the intended use and changes in the analysis algorithm were evaluated by SA. The average sensitivity, specificity, and AUC were 93% (minimum 91.1, maximum 97), 89.6% (minimum 85.9, maximum 96), and 0.96 (minimum 0.96, maximum 0.97), respectively. The average interval between applications was 348 days (minimum –18, maximum 975), which showed that the improvements were implemented within approximately one year. This is the first comprehensive study on AI/ML-based CAD products that have been improved post-market to elucidate evaluation points for post-market improvements. The findings will be informative for the industry and academia in developing and improving AI/ML-based CAD. Computer-aided detection (CADe), computer-aided diagnosis (CADx), and computer-aided simple triage (CAST) utilising artificial intelligence (AI) and machine learning (ML) are under development around the world. AI/ML-based CAD is continually undergoing post-market improvement using real-world data collected from clinical sites. Therefore, regulatory authorities such as the U.S. Food and Drug Administration (FDA), the Ministry of Health, Labour, and Welfare (MHLW) and the Pharmaceuticals and Medical Device Agency (PMDA) in Japan, and the European Medicines Agency (EMA) in Europe are investigating appropriate regulation systems for AI features. Here, we investigated AI/ML-based CAD products that have been improved post-market by surveying a product code database published by the FDA. Eight products that have been modified after marketing were identified. We found that post-market improvements that did not change the intended use were approved after evaluation by a standalone software test. The findings will be informative for the industry and academia in developing and improving AI/ML-based CAD.

Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products

Performance evaluation methods for improvements at post-market of artificial intelligence/machine learning-based computer-aided detection/diagnosis/triage in the United States

Systematic analysis of the test design and performance of AI/ML-based medical devices approved for triage/detection/diagnosis in the USA and Japan

Computer-aided Detection for Virtual Colonoscopy

Overview of Computer-aided Diagnosis in Mammographic Imaging

Insights into post-marketing clinical validation of companion diagnostics with reference to the FDA, EMA, PMDA, and MFDS

S304 Standardizing Criteria for the Comparison of AI-Based Computer-Aided Detection Programs in Colonoscopy: A Modified Delphi Approach

A case study regarding clinical performance evaluation method of medical device software for approval

Development and Application of a Standardized Testset for an Artificial Intelligence Medical Device Intended for the Computer-Aided Diagnosis of Diabetic Retinopathy

Developing Interactive Computer-Aided Detection Tools to Support Translational Clinical Research

Artificial Intelligence in Breast Cancer Screening: Evaluation of FDA Device Regulation and Future Recommendations

Computer-aided diagnosis in medical imaging: Historical review, current status and future potential

Harmonizing Safety and Speed: A Human-Algorithm Approach to Enhance the FDA's Medical Device Clearance Policy

FDA's perspectives on cardiovascular devices

Digital Tools-Regulatory Considerations for Application in Clinical Trials

Evaluating the Safety Profile of Non-Active Implantable Medical Devices Compared with Medicines

Clinical Application of Computer-Aided Diagnosis System in Breast Ultrasound: A Prospective Multicenter Study.

Evolution of China regulatory guidance on the clinical evaluation of medical devices and its implication on pre-market and post-approval clinical evaluation strategies

Evaluation of the Standard Test Methods for Optical Coherence Tomograph for the Posterior Segment of the Human Eye

Discussions on the standardized safety evaluation of new drugs in phase I clinical trials