Assessing computational predictions of antimicrobial resistance phenotypes from microbial genomes

Kaixin Hu,Fernando Meyer,Zhi-Luo Deng,Ehsaneddin Asgari,Tzu-Hao Kuo,Philipp C. Münch,Alice C. McHardy
DOI: https://doi.org/10.1101/2024.01.31.578169
2024-03-09
Abstract:The advent of rapid whole-genome sequencing has created new opportunities for computational prediction of antimicrobial resistance (AMR) phenotypes from genomic data. Both rule-based and machine learning (ML) approaches have been explored for this task, but systematic benchmarking is still needed. Here, we evaluated four state-of-the-art ML methods (Kover, PhenotypeSeeker, Seq2Geno2Pheno, and Aytan-Aktug), an ML baseline, and the rule-based ResFinder by training and testing each of them across 78 species–antibiotic datasets, using a rigorous benchmarking workflow that integrates three evaluation approaches, each paired with three distinct sample splitting methods. Our analysis revealed considerable variation in the performance across techniques and datasets. Whereas ML methods generally excelled for closely related strains, ResFinder excelled for handling divergent genomes. Overall, Kover most frequently ranked top among the ML approaches, followed by PhenotypeSeeker and Seq2Geno2Pheno. AMR phenotypes for antibiotic classes such as macrolides and sulfonamides were predicted with the highest accuracies. The quality of predictions varied substantially across species–antibiotic combinations, particularly for beta-lactams; across species, resistance phenotyping of the beta-lactams compound, aztreonam, amox-clav, cefoxitin, ceftazidime, and piperacillin/tazobactam, alongside tetracyclines demonstrated more variable performance than the other benchmarked antibiotics. By organism, and phenotypes were more robustly predicted than those of , , , , , , , , and Mycobacterium tuberculos . In addition, our study provides software recommendations for each species–antibiotic combination. It furthermore highlights the need for optimization for robust clinical applications, particularly for strains that diverge substantially from those used for training.
Bioinformatics
What problem does this paper attempt to address?
This paper evaluates the performance of the latest machine learning (ML) methods and rule-based methods for predicting antimicrobial resistance phenotypes from microbial genomic data. The study analyzes four advanced ML methods (Kover, PhenotypeSeeker, Seq2Geno2Pheno, and Aytan-Aktug), an ML baseline, and the rule-based ResFinder using a rigorous benchmark testing process that includes different evaluation methods and sample splitting strategies. The results show significant variations in the performance of different techniques across different species and antibiotic datasets. ML methods perform well in closely related strains, while ResFinder excels in handling genomes with larger variations. Kover consistently ranks the highest among the ML methods, followed by PhenotypeSeeker and Seq2Geno2Pheno. The paper emphasizes the importance of optimizing these methods for clinical applications, particularly for strains that significantly differ from the training samples.