Retrospective validation of MetaSystems’ deep-learning-based digital microscopy platform with assistance compared to manual fluorescence microscopy for detection of mycobacteria

Claudine Desruisseaux,Conor Broderick,Valéry Lavergne,Kim Sy,Duang-Jai Garcia,Gaurav Barot,Kerstin Locher,Charlene Porter,Mélissa Caza,Marthe K. Charles
DOI: https://doi.org/10.1128/jcm.01069-23
2024-02-01
Journal of Clinical Microbiology
Abstract:ABSTRACT This study aimed to validate Metasystems’ automated acid-fast bacilli (AFB) smear microscopy scanning and deep-learning-based image analysis module (Neon Metafer) with assistance on respiratory and pleural samples, compared to conventional manual fluorescence microscopy (MM). Analytical parameters were assessed first, followed by a retrospective validation study. In all, 320 archived auramine-O-stained slides selected non-consecutively [85 originally reported as AFB-smear-positive, 235 AFB-smear-negative slides; with an overall mycobacterial culture positivity rate of 24.1% (77/320)] underwent whole-slide imaging and were analyzed by the Metafer Neon AFB Module (version 4.3.130) using a predetermined probability threshold (PT) for AFB detection of 96%. Digital slides were then examined by a trained reviewer blinded to previous AFB smear and culture results, for the final interpretation of assisted digital microscopy (a-DM). Paired results from both microscopic methods were compared to mycobacterial culture. A scanning failure rate of 10.6% (34/320) was observed, leaving 286 slides for analysis. After discrepant analysis, concordance, positive and negative agreements were 95.5% (95%CI, 92.4%–97.6%), 96.2% (95%CI, 89.2%–99.2%), and 95.2% (95%CI, 91.3%–97.7%), respectively. Using mycobacterial culture as reference standard, a-DM and MM had comparable sensitivities: 90.7% (95%CI, 81.7%–96.2%) versus 92.0% (95%CI, 83.4%–97.0%) ( P -value = 1.00); while their specificities differed 91.9% (95%CI, 87.4%–95.2%) versus 95.7% (95%CI, 92.1%–98.0%), respectively ( P -value = 0.03). Using a PT of 96%, MetaSystems’ platform shows acceptable performance. With a national laboratory staff shortage and a local low mycobacterial infection rate, this instrument when combined with culture, can reliably triage-negative AFB-smear respiratory slides and identify positive slides requiring manual confirmation and semi-quantification. IMPORTANCE This manuscript presents a full validation of MetaSystems’ automated acid-fast bacilli (AFB) smear microscopy scanning and deep-learning-based image analysis module using a probability threshold of 96% including accuracy, precision studies, and evaluation of limit of AFB detection on respiratory samples when the technology is used with assistance. This study is complementary to the conversation started by Tomasello et al. on the use of image analysis artificial intelligence software in routine mycobacterial diagnostic activities within the context of high-throughput laboratories with low incidence of tuberculosis.
microbiology
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: to verify how the MetaSystems' deep - learning - based digital microscope platform (Neon Metafer) performs when detecting acid - fast bacilli (AFB) smears compared with the traditional manual fluorescence microscope (MM). Specifically, the study aims to evaluate the performance of this platform in the automated scanning and image analysis modules and explore whether it can replace the traditional method in the routine mycobacterium detection workflow. ### Research Background - **Early Accurate Detection**: Rapid and accurate detection of mycobacterium infections (such as tuberculosis) is crucial for clinical management, treatment decision - making, and infection prevention and control. - **Limitations of Traditional Methods**: Despite significant progress in molecular diagnostic techniques, the manual fluorescence microscope (MM) is still a commonly used detection method in high - and low - incidence tuberculosis environments. However, this method is time - consuming and depends on the operator's experience, and has certain limitations. ### Research Objectives 1. **Primary Objectives**: - Evaluate the analytical performance of the MetaSystems platform on the automated scanning and deep - learning image analysis modules. - Compare the diagnostic consistency and accuracy of Metafer software - assisted digital microscope (a - DM) and manual fluorescence microscope (MM) in AFB smears of respiratory and pleural samples. 2. **Secondary Objectives**: - Evaluate the reliability of the software's ability to grade AFB scores. ### Method Overview - **Sample Selection**: 320 archived acid - fast - stained slides were selected for a retrospective validation study. - **Image Analysis**: The deep neural network (DNN) algorithm of Metafer software was used to classify the image of each slide and make judgments according to the preset probability threshold (PT = 96%). - **Result Comparison**: The results of a - DM and MM were compared with mycobacterium culture as the reference standard to evaluate consistency, sensitivity, and specificity. ### Key Findings - **Overall Consistency**: After inconsistent analysis, the overall consistency between a - DM and MM was 95.5% (95% CI, 92.4%–97.6%), the positive consistency rate was 96.2% (95% CI, 89.2%–99.2%), and the negative consistency rate was 95.2% (95% CI, 91.3%–97.7%). - **Sensitivity and Specificity**: Taking mycobacterium culture as the reference standard, the sensitivity of a - DM was 90.7% (95% CI, 81.7%–96.2%), and the specificity was 91.9% (95% CI, 87.4%–95.2%). In comparison, the sensitivity of MM was 92.0% (95% CI, 83.4%–97.0%), and the specificity was 95.7% (95% CI, 92.1%–98.0%). There was no statistically significant difference in sensitivity between the two (P - value = 1.00), but the difference in specificity was significant (P - value = 0.03). ### Conclusion The MetaSystems' platform shows acceptable performance when PT is set to 96%. Combined with the culture results, this instrument can reliably screen negative AFB smears and identify positive slides that require manual confirmation and semi - quantification in cases of national laboratory staff shortages and low local mycobacterium infection rates. Based on these research results, the author believes that a - DM can be an effective alternative tool to traditional MM, especially in high - throughput laboratories, which helps improve work efficiency and reduce the artificial burden.