Ruchir Rastogi,Ryan Chung,Sindy Li,Chang Li,Kyoungyeul Lee,Junwoo Woo,Dong-Wook Kim,Changwon Keum,Giulia Babbi,Pier Luigi Martelli,Castrense Savojardo,Rita Casadio,Kirsley Chennen,Thomas Weber,Olivier Poch,Francois Ancien,Gabriel Cia,Fabrizio Pucci,Daniele Raimondi,Wim Vranken,Marianne Rooman,Celine Marquet,Tobias Olenyi,Burkhard Rost,Gaia Andreoletti,Akash Kamandula,Yisu Peng,Constantina Bakolitsa,Matthew Mort,David N. Cooper,Timothy Bergquist,Vikas Pejaver,Xiaoming Liu,Predrag Radivojac,Steven E. Brenner,Nilah M. Ioannidis

Abstract:Regular, systematic, and independent assessment of computational tools used to predict the pathogenicity of missense variants is necessary to evaluate their clinical and research utility and suggest directions for future improvement. Here, as part of the sixth edition of the Critical Assessment of Genome Interpretation (CAGI) challenge, we assess missense variant effect predictors (or variant impact predictors) on an evaluation dataset of rare missense variants from disease-relevant databases. Our assessment evaluates predictors submitted to the CAGI6 Annotate-All-Missense challenge, predictors commonly used by the clinical genetics community, and recently developed deep learning methods for variant effect prediction. To explore a variety of settings that are relevant for different clinical and research applications, we assess performance within different subsets of the evaluation data and within high-specificity and high-sensitivity regimes. We find strong performance of many predictors across multiple settings. Meta-predictors tend to outperform their constituent individual predictors; however, several individual predictors have performance similar to that of commonly used meta-predictors. The relative performance of predictors differs in high-specificity and high-sensitivity regimes, suggesting that different methods may be best suited to different use cases. We also characterize two potential sources of bias. Predictors that incorporate allele frequency as a predictive feature tend to have reduced performance when distinguishing pathogenic variants from very rare benign variants, and predictors supervised on pathogenicity labels from curated variant databases often learn label imbalances within genes. Overall, we find notable advances over the oldest and most cited missense variant effect predictors and continued improvements among the most recently developed tools, and the CAGI Annotate-All-Missense challenge (also termed the Missense Marathon) will continue to assess state-of-the-art methods as the field progresses. Together, our results help illuminate the current clinical and research utility of missense variant effect predictors and identify potential areas for future development.

Assessing predictions on fitness effects of missense variants in HMBS in CAGI6

Critical assessment of missense variant effect predictors on disease-relevant variant data

Evaluation of in silico predictors on short nucleotide variants in HBA1, HBA2, and HBB associated with haemoglobinopathies

Accurate proteome-wide missense variant effect prediction with AlphaMissense

MMPatho: Leveraging Multilevel Consensus and Evolutionary Information for Enhanced Missense Mutation Pathogenic Prediction

FiTMuSiC: leveraging structural and (co)evolutionary data for protein fitness prediction

Evaluation of enzyme activity predictions for variants of unknown significance in Arylsulfatase A

Assessing Predictions Of The Impact Of Variants On Splicing In Cagi5

Exploring the effects of missense mutations on protein thermodynamics through structure-based approaches: findings from the CAGI6 challenges

Rapid discrimination between deleterious and benign missense mutations in the CAGI 6 experiment

TargetMM: Accurate Missense Mutation Prediction by Utilizing Local and Global Sequence Information with Classifier Ensemble.

Amendment history : Corrigendum ( April 2017 ) A modifier screen identifies DNAJB 6 as a cardiomyopathy susceptibility gene

Using computational approaches to enhance the interpretation of missense variants in the PAX6 gene

The landscape of regional missense mutational intolerance quantified from 125,748 exomes

Assessment of Computational Methods for Predicting the Effects of Missense Mutations in Human Cancers.

Assessment of Predicted Enzymatic Activity of Α‐ N ‐acetylglucosaminidase Variants of Unknown Significance for CAGI 2016

Evaluating predictors of kinase activity of STK11 variants identified in primary human non-small cell lung cancers

ProteoMutaMetrics: machine learning approaches for solute carrier family 6 mutation pathogenicity prediction

Predicting Pathology of Missense Mutations through Protein-Specific Evolutionary Pattern

Community-wide evaluation of methods for predicting the effect of mutations on protein-protein interactions.

Accurate Sequence-Based Prediction of Deleterious Nssnps with Multiple Sequence Profiles and Putative Binding Residues