Ruchir Rastogi,Ryan Chung,Sindy Li,Chang Li,Kyoungyeul Lee,Junwoo Woo,Dong-Wook Kim,Changwon Keum,Giulia Babbi,Pier Luigi Martelli,Castrense Savojardo,Rita Casadio,Kirsley Chennen,Thomas Weber,Olivier Poch,Francois Ancien,Gabriel Cia,Fabrizio Pucci,Daniele Raimondi,Wim Vranken,Marianne Rooman,Celine Marquet,Tobias Olenyi,Burkhard Rost,Gaia Andreoletti,Akash Kamandula,Yisu Peng,Constantina Bakolitsa,Matthew Mort,David N. Cooper,Timothy Bergquist,Vikas Pejaver,Xiaoming Liu,Predrag Radivojac,Steven E. Brenner,Nilah M. Ioannidis

Abstract:Regular, systematic, and independent assessment of computational tools used to predict the pathogenicity of missense variants is necessary to evaluate their clinical and research utility and suggest directions for future improvement. Here, as part of the sixth edition of the Critical Assessment of Genome Interpretation (CAGI) challenge, we assess missense variant effect predictors (or variant impact predictors) on an evaluation dataset of rare missense variants from disease-relevant databases. Our assessment evaluates predictors submitted to the CAGI6 Annotate-All-Missense challenge, predictors commonly used by the clinical genetics community, and recently developed deep learning methods for variant effect prediction. To explore a variety of settings that are relevant for different clinical and research applications, we assess performance within different subsets of the evaluation data and within high-specificity and high-sensitivity regimes. We find strong performance of many predictors across multiple settings. Meta-predictors tend to outperform their constituent individual predictors; however, several individual predictors have performance similar to that of commonly used meta-predictors. The relative performance of predictors differs in high-specificity and high-sensitivity regimes, suggesting that different methods may be best suited to different use cases. We also characterize two potential sources of bias. Predictors that incorporate allele frequency as a predictive feature tend to have reduced performance when distinguishing pathogenic variants from very rare benign variants, and predictors supervised on pathogenicity labels from curated variant databases often learn label imbalances within genes. Overall, we find notable advances over the oldest and most cited missense variant effect predictors and continued improvements among the most recently developed tools, and the CAGI Annotate-All-Missense challenge (also termed the Missense Marathon) will continue to assess state-of-the-art methods as the field progresses. Together, our results help illuminate the current clinical and research utility of missense variant effect predictors and identify potential areas for future development.

CADD v1.7: using protein language models, regulatory CNNs and other nucleotide-level scores to improve genome-wide variant predictions

varCADD: large sets of standing genetic variation enable genome-wide pathogenicity prediction

A generic pipeline for CADD score generation: chickenCADD and turkeyCADD

CADD-Splice—improving genome-wide variant effect prediction using deep learning-derived splice scores

A general framework for estimating the relative pathogenicity of human genetic variants

DANN: a deep learning approach for annotating the pathogenicity of genetic variants

Computational Assessment of the Regulation-Modulating Potential for Noncoding Variants

Cross-protein transfer learning substantially improves disease variant prediction

Improving estimates of negative selection in human genome using CAPS

Exploring Functional Variant Discovery in Non-Coding Regions with Sinbad

GAPIT Version 2: an Enhanced Integrated Tool for Genomic Association and Prediction

Rapid discrimination between deleterious and benign missense mutations in the CAGI 6 experiment

A consensus variant-to-function score to functionally prioritize variants for disease

De novo pattern discovery enables robust assessment of functional consequences of non-coding variants

Genome-wide prediction of disease variant effects with a deep protein language model

VariPred: Enhancing Pathogenicity Prediction of Missense Variants Using Protein Language Models

A method for scoring the cell type-specific impacts of noncoding variants in personal genomes

Computational Assessment of the Expression-modulating Potential for Non-coding Variants.

Critical assessment of missense variant effect predictors on disease-relevant variant data

Multimodal learning of noncoding variant effects using genome sequence and chromatin structure

Fine-tuning Protein Language Models with Deep Mutational Scanning improves Variant Effect Prediction