Viral transmission: Deadly contact

K. Bussell

DOI: https://doi.org/10.1038/nrm1081

2003-04-01

Abstract:

What problem does this paper attempt to address?

Expert-guided protein Language Models enable accurate and blazingly fast fitness prediction

Céline Marquet,Julius Schlensok,Marina Abakarova,Burkhard Rost,Elodie Laine

DOI: https://doi.org/10.1101/2024.04.24.590982

2024-09-09

Abstract:Exhaustive experimental annotation of the effect of all known protein variants remains daunting and expensive, stressing the need for scalable effect predictions. We introduce VespaG, a blazingly fast missense amino acid variant effect predictor, leveraging protein Language Model (pLM) embeddings as input to a minimal deep learning model. To overcome the sparsity of experimental training data, we created a dataset of 39 million single amino acid variants from the human proteome applying the multiple sequence alignment-based effect predictor GEMME as a pseudo standard-of-truth. This setup increases interpretability compared to the baseline pLM and is easily retrainable with novel or updated pLMs. Assessed against the ProteinGym benchmark (217 multiplex assays of variant effect - MAVE - with 2.5 million variants), VespaG achieved a mean Spearman correlation of 0.48±0.02, matching top-performing methods evaluated on the same data. VespaG has the advantage of being orders of magnitude faster, predicting all mutational landscapes of all proteins in proteomes such as or in under 30 minutes on a consumer laptop (12-core CPU, 16 GB RAM).

Bioinformatics
Modeling the language of life – Deep Learning Protein Sequences

Michael Heinzinger,Ahmed Elnaggar,Yu Wang,Christian Dallago,Dmitrii Nechaev,Florian Matthes,Burkhard Rost

DOI: https://doi.org/10.1101/614313

2019-04-19

Abstract:Abstract Background One common task in Computational Biology is the prediction of aspects of protein function and structure from their amino acid sequence. For 26 years, most state-of-the-art approaches toward this end have been marrying machine learning and evolutionary information. The retrieval of related proteins from ever growing sequence databases is becoming so time-consuming that the analysis of entire proteomes becomes challenging. On top, evolutionary information is less powerful for small families, e.g. for proteins from the Dark Proteome . Results We introduce a novel way to represent protein sequences as continuous vectors ( embeddings ) by using the deep bi-directional model ELMo taken from natural language processing (NLP). The model has effectively captured the biophysical properties of protein sequences from unlabeled big data (UniRef50). After training, this knowledge is transferred to single protein sequences by predicting relevant sequence features. We refer to these new embeddings as SeqVec ( Seq uence-to- Vec tor) and demonstrate their effectiveness by training simple convolutional neural networks on existing data sets for two completely different prediction tasks. At the per-residue level, we significantly improved secondary structure (for NetSurfP-2.0 data set: Q3=79%±1, Q8=68%±1) and disorder predictions (MCC=0.59±0.03) over methods not using evolutionary information. At the per-protein level, we predicted subcellular localization in ten classes (for DeepLoc data set: Q10=68%±1) and distinguished membrane-bound from water-soluble proteins (Q2= 87%±1). All results built upon the embeddings gained from the new tool SeqVec neither explicitly nor implicitly using evolutionary information. Nevertheless, it improved over some methods using such information. Where the lightning-fast HHblits needed on average about two minutes to generate the evolutionary information for a target protein, SeqVec created the vector representation on average in 0.03 seconds. Conclusion We have shown that transfer learning can be used to capture biochemical or biophysical properties of protein sequences from large unlabeled sequence databases. The effectiveness of the proposed approach was showcased for different prediction tasks using only single protein sequences. SeqVec embeddings enable predictions that outperform even some methods using evolutionary information. Thus, they prove to condense the underlying principles of protein sequences. This might be the first step towards competitive predictions based only on single protein sequences. Availability SeqVec: https://github.com/mheinzinger/SeqVec Prediction server: https://embed.protein.properties
From a single sequence to evolutionary trajectories: protein language models capture the evolutionary potential of SARS-CoV-2 protein sequences

Kieran D. Lamb,Joseph Hughes,Spyros Lytras,Francesca Young,Orges Koci,James Herzig,Simon C Lovell,Joe Grove,Ke Yuan,David L. Robertson

DOI: https://doi.org/10.1101/2024.07.05.602129

2024-09-18

Abstract:Protein language models (PLMs) capture features of protein three-dimensional structure from amino acid sequences alone, without requiring multiple sequence alignments (MSA). The concepts of grammar and semantics from natural language have the potential to capture functional properties of proteins. Here, we investigate how these representations enable assessment of variation due to mutation. Applied to SARS-CoV-2’s spike protein using in silico deep mutational scanning (DMS) we demonstrate the PLM, ESM-2, has learned the sequence context within which variation occurs, capturing evolutionary constraint. This recapitulates what conventionally requires MSA data to predict. Unlike other state-of-the-art methods which require protein structures or multiple sequences for training, we show what can be accomplished using an unmodified pretrained PLM. We demonstrate that the grammaticality and semantic scores represent novel metrics. Applied to SARS-CoV-2 variants across the pandemic we show that ESM-2 representations encode the evolutionary history between variants, as well as the distinct nature of variants of concern upon their emergence, associated with shifts in receptor binding and antigenicity. PLM likelihoods can also identify epistatic interactions among sites in the protein. Our results here affirm that PLMs are broadly useful for variant-effect prediction, including unobserved changes, and can be applied to understand novel viral pathogens with the potential to be applied to any protein sequence, pathogen or otherwise.

Bioinformatics
Assessing the role of evolutionary information for enhancing protein language model embeddings

Kyra Erckert,Burkhard Rost

DOI: https://doi.org/10.1038/s41598-024-71783-8

2024-09-05

Abstract:Embeddings from protein Language Models (pLMs) are replacing evolutionary information from multiple sequence alignments (MSAs) as the most successful input for protein prediction. Is this because embeddings capture evolutionary information? We tested various approaches to explicitly incorporate evolutionary information into embeddings on various protein prediction tasks. While older pLMs (SeqVec, ProtBert) significantly improved through MSAs, the more recent pLM ProtT5 did not benefit. For most tasks, pLM-based outperformed MSA-based methods, and the combination of both even decreased performance for some (intrinsic disorder). We highlight the effectiveness of pLM-based methods and find limited benefits from integrating MSAs.
Fine-tuning Protein Language Models with Deep Mutational Scanning improves Variant Effect Prediction

Aleix Lafita,Ferran Gonzalez,Mahmoud Hossam,Paul Smyth,Jacob Deasy,Ari Allyn-Feuer,Daniel Seaton,Stephen Young

2024-05-10

Abstract:Protein Language Models (PLMs) have emerged as performant and scalable tools for predicting the functional impact and clinical significance of protein-coding variants, but they still lag experimental accuracy. Here, we present a novel fine-tuning approach to improve the performance of PLMs with experimental maps of variant effects from Deep Mutational Scanning (DMS) assays using a Normalised Log-odds Ratio (NLR) head. We find consistent improvements in a held-out protein test set, and on independent DMS and clinical variant annotation benchmarks from ProteinGym and ClinVar. These findings demonstrate that DMS is a promising source of sequence diversity and supervised training data for improving the performance of PLMs for variant effect prediction.

Genomics,Machine Learning
Alignment-free estimation of sequence conservation for identifying functional sites using protein sequence embeddings

Wayland Yeung,Zhongliang Zhou,Sheng Li,Natarajan Kannan

DOI: https://doi.org/10.1093/bib/bbac599

IF: 9.5

2023-01-12

Briefings in Bioinformatics

Abstract:Protein language modeling is a fast-emerging deep learning method in bioinformatics with diverse applications such as structure prediction and protein design. However, application toward estimating sequence conservation for functional site prediction has not been systematically explored. Here, we present a method for the alignment-free estimation of sequence conservation using sequence embeddings generated from protein language models. Comprehensive benchmarks across publicly available protein language models reveal that ESM2 models provide the best performance to computational cost ratio for conservation estimation. Applying our method to full-length protein sequences, we demonstrate that embedding-based methods are not sensitive to the order of conserved elements—conservation scores can be calculated for multidomain proteins in a single run, without the need to separate individual domains. Our method can also identify conserved functional sites within fast-evolving sequence regions (such as domain inserts), which we demonstrate through the identification of conserved phosphorylation motifs in variable insert segments in protein kinases. Overall, embedding-based conservation analysis is a broadly applicable method for identifying potential functional sites in any full-length protein sequence and estimating conservation in an alignment-free manner. To run this on your protein sequence of interest, try our scripts at https://github.com/esbgkannan/kibby.

biochemical research methods,mathematical & computational biology
Protein embeddings and deep learning predict binding residues for various ligand classes

Maria Littmann,Michael Heinzinger,Christian Dallago,Konstantin Weissenow,Burkhard Rost

DOI: https://doi.org/10.1038/s41598-021-03431-4

2021-12-13

Abstract:One important aspect of protein function is the binding of proteins to ligands, including small molecules, metal ions, and macromolecules such as DNA or RNA. Despite decades of experimental progress many binding sites remain obscure. Here, we proposed bindEmbed21, a method predicting whether a protein residue binds to metal ions, nucleic acids, or small molecules. The Artificial Intelligence (AI)-based method exclusively uses embeddings from the Transformer-based protein Language Model (pLM) ProtT5 as input. Using only single sequences without creating multiple sequence alignments (MSAs), bindEmbed21DL outperformed MSA-based predictions. Combination with homology-based inference increased performance to F1 = 48 ± 3% (95% CI) and MCC = 0.46 ± 0.04 when merging all three ligand classes into one. All results were confirmed by three independent data sets. Focusing on very reliably predicted residues could complement experimental evidence: For the 25% most strongly predicted binding residues, at least 73% were correctly predicted even when ignoring the problem of missing experimental annotations. The new method bindEmbed21 is fast, simple, and broadly applicable-neither using structure nor MSAs. Thereby, it found binding residues in over 42% of all human proteins not otherwise implied in binding and predicted about 6% of all residues as binding to metal ions, nucleic acids, or small molecules.
Synthesis of inorganic polymers as glass precursors and for other uses: Pre‐ceramic block or graft copolymers as potential precursors to composite materials

Kay A. Youngdahl,M. L. Hoppe,R. Laine,J. A. Rahn,J. Harrod

DOI: https://doi.org/10.1002/AOC.590070808

1993-12-01

Abstract:Abstract : Preceramic block or graft copolymers may offer entree into nanocomposite ceramics provided the two homopolymers are immiscible and one can carefully control the size of the blocks or grafts. We are exploring the possibility of making copolymers from methlsilsesquioxane, -(MeS(O)1.5)(x-), (SiO), a precursor to black glass and the polysilazane, -(MeHSiNH)(x-), (SiMe), (SiMe), a precursor to silicon carbide nitride. Our initial efforts have been directed towards delineating the chemical transformations that SiO, prepared by room temperature catalytic redistribution of -(MeHSiO)(x-) using Cp2 TiMe2 as the catalyst (0.1 wt%), undergoes as it is heated to 900 C in N2. We find that although Cp2TiMe2 will not catalyze the redistribution of SiMe at room temperature; in the presence of even small amounts of -(MeHSiO) (x-), it is an active catalyst precursor and copolymer be formed. Spectral and chemical composition studies on the pyrolysis products of the copolymers and SiO are described.
Updated benchmarking of variant effect predictors using deep mutational scanning

Benjamin J Livesey,Joseph A Marsh

DOI: https://doi.org/10.15252/msb.202211474

2023-08-08

Abstract:The assessment of variant effect predictor (VEP) performance is fraught with biases introduced by benchmarking against clinical observations. In this study, building on our previous work, we use independently generated measurements of protein function from deep mutational scanning (DMS) experiments for 26 human proteins to benchmark 55 different VEPs, while introducing minimal data circularity. Many top-performing VEPs are unsupervised methods including EVE, DeepSequence and ESM-1v, a protein language model that ranked first overall. However, the strong performance of recent supervised VEPs, in particular VARITY, shows that developers are taking data circularity and bias issues seriously. We also assess the performance of DMS and unsupervised VEPs for discriminating between known pathogenic and putatively benign missense variants. Our findings are mixed, demonstrating that some DMS datasets perform exceptionally at variant classification, while others are poor. Notably, we observe a striking correlation between VEP agreement with DMS data and performance in identifying clinically relevant variants, strongly supporting the validity of our rankings and the utility of DMS for independent benchmarking.
TransEFVP: A Two-Stage Approach for the Prediction of Human Pathogenic Variants Based on Protein Sequence Embedding Fusion

Zihao Yan,Fang Ge,Yan Liu,Yumeng Zhang,Fuyi Li,Jiangning Song,Dong-Jun Yu

DOI: https://doi.org/10.1021/acs.jcim.3c02019

2024-02-26

Abstract:Studying the effect of single amino acid variations (SAVs) on protein structure and function is integral to advancing our understanding of molecular processes, evolutionary biology, and disease mechanisms. Screening for deleterious variants is one of the crucial issues in precision medicine. Here, we propose a novel computational approach, TransEFVP, based on large-scale protein language model embeddings and a transformer-based neural network to predict disease-associated SAVs. The model adopts a two-stage architecture: the first stage is designed to fuse different feature embeddings through a transformer encoder. In the second stage, a support vector machine model is employed to quantify the pathogenicity of SAVs after dimensionality reduction. The prediction performance of TransEFVP on blind test data achieves a Matthews correlation coefficient of 0.751, an F1-score of 0.846, and an area under the receiver operating characteristic curve of 0.871, higher than the existing state-of-the-art methods. The benchmark results demonstrate that TransEFVP can be explored as an accurate and effective SAV pathogenicity prediction method. The data and codes for TransEFVP are available at https://github.com/yzh9607/TransEFVP/tree/master for academic use.
Genome-wide prediction of disease variant effects with a deep protein language model

Nadav Brandes,Grant Goldman,Charlotte H. Wang,Chun Jimmie Ye,Vasilis Ntranos

DOI: https://doi.org/10.1038/s41588-023-01465-0

IF: 30.8

2023-08-11

Nature Genetics

Abstract:Predicting the effects of coding variants is a major challenge. While recent deep-learning models have improved variant effect prediction accuracy, they cannot analyze all coding variants due to dependency on close homologs or software limitations. Here we developed a workflow using ESM1b, a 650-million-parameter protein language model, to predict all ~450 million possible missense variant effects in the human genome, and made all predictions available on a web portal. ESM1b outperformed existing methods in classifying ~150,000 ClinVar/HGMD missense variants as pathogenic or benign and predicting measurements across 28 deep mutational scan datasets. We further annotated ~2 million variants as damaging only in specific protein isoforms, demonstrating the importance of considering all isoforms when predicting variant effects. Our approach also generalizes to more complex coding variants such as in-frame indels and stop-gains. Together, these results establish protein language models as an effective, accurate and general approach to predicting variant effects.

genetics & heredity
Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations

Benjamin J Livesey,Joseph A Marsh

DOI: https://doi.org/10.15252/msb.20199380

IF: 13.068

2020-07-01

Molecular Systems Biology

Abstract:<p>To deal with the huge number of novel protein‐coding variants identified by genome and exome sequencing studies, many computational variant effect predictors (VEPs) have been developed. Such predictors are often trained and evaluated using different variant data sets, making a direct comparison between VEPs difficult. In this study, we use 31 previously published deep mutational scanning (DMS) experiments, which provide quantitative, independent phenotypic measurements for large numbers of single amino acid substitutions, in order to benchmark and compare 46 different VEPs. We also evaluate the ability of DMS measurements and VEPs to discriminate between pathogenic and benign missense variants. We find that DMS experiments tend to be superior to the top‐ranking predictors, demonstrating the tremendous potential of DMS for identifying novel human disease mutations. Among the VEPs, DeepSequence clearly stood out, showing both the strongest correlations with DMS data and having the best ability to predict pathogenic mutations, which is especially remarkable given that it is an unsupervised method. We further recommend SNAP2, DEOGEN2, SNPs&GO, SuSPect and REVEL based upon their performance in these analyses.</p>

biochemistry & molecular biology
Enhancing predictions of protein stability changes induced by single mutations using MSA-based language models

Francesca Cuturello,Marco Celoria,Alessio Ansuini,Alberto Cazzaniga

DOI: https://doi.org/10.1093/bioinformatics/btae447

IF: 5.8

2024-07-01

Bioinformatics

Abstract:Abstract Motivation Protein language models offer a new perspective for addressing challenges in structural biology, while relying solely on sequence information. Recent studies have investigated their effectiveness in forecasting shifts in thermodynamic stability caused by single amino acid mutations, a task known for its complexity due to the sparse availability of data, constrained by experimental limitations. To tackle this problem, we introduce two key novelties: leveraging a protein language model that incorporates Multiple Sequence Alignments to capture evolutionary information, and using a recently released mega-scale dataset with rigorous data preprocessing to mitigate overfitting. Results We ensure comprehensive comparisons by fine-tuning various pretrained models, taking advantage of analyses such as ablation studies and baselines evaluation. Our methodology introduces a stringent policy to reduce the widespread issue of data leakage, rigorously removing sequences from the training set when they exhibit significant similarity with the test set. The MSA Transformer emerges as the most accurate among the models under investigation, given its capability to leverage co-evolution signals encoded in aligned homologous sequences. Moreover, the optimized MSA Transformer outperforms existing methods and exhibits enhanced generalization power, leading to a notable improvement in predicting changes in protein stability resulting from point mutations. Availability and implementation Code and data at https://github.com/RitAreaSciencePark/PLM4Muts.

biochemical research methods,biotechnology & applied microbiology,mathematical & computational biology
Benchmarking text-integrated protein language model embeddings and embedding fusion on diverse downstream tasks

Young Su Ko,Jonathan Parkinson,Wei Wang

DOI: https://doi.org/10.1101/2024.08.24.609531

2024-08-26

Abstract:Protein language models (pLMs) have traditionally been trained in an unsupervised manner using large protein sequence databases with an autoregressive or masked-language modeling training paradigm. Recent methods have attempted to enhance pLMs by integrating additional information, in the form of text, which are referred to as “text+protein” language models (tpLMs). We evaluate and compare six tpLMs (OntoProtein, ProteinDT, ProtST, ProteinCLIP, ProTrek, and ESM3) against ESM2, a baseline text-free pLM, across six downstream tasks designed to assess the learned protein representations. We find that while tpLMs outperform ESM2 in five out of six benchmarks, no tpLM was consistently the best. Thus, we additionally investigate the potential of embedding fusion, exploring whether the combinations of tpLM embeddings can improve performance on the benchmarks by exploiting the strengths of multiple tpLMs. We find that combinations of tpLM embeddings outperform single tpLM embeddings in five out of six benchmarks, highlighting its potential as a useful strategy in the field of machine-learning for proteins. To facilitate the practical application of embedding fusion, we outline a heuristic framework to efficiently identify the optimal combination of embeddings, reducing the exponential time complexity of an exhaustive combination search down to a manageable linear time complexity. Using our embedding fusion framework, we achieve state-of-the-art performances on the protein-protein interaction prediction and homologous sequence recovery tasks without any specific model adjustments or hyperparameter tuning. Our experiments suggest embedding fusion is a useful tool in the machine-learning for proteins toolbox. Lastly, this study highlights the potential of future research on additional strategies for maximizing the utility of pLMs.

Bioinformatics
Efficient Inference, Training, and Fine-tuning of Protein Language Models

Muhammed Hasan Celik,Xiaohui Xie

DOI: https://doi.org/10.1101/2024.10.22.619563

2024-10-25

Abstract:Protein language models have shown great promise in predicting protein structure, function, and the effects of missense variants on protein fitness. However, their use has been limited by the substantial computational resources required. In this work, we focus on improving the computational efficiency of protein language models (PLMs), specifically the Evolutionary Scale Modeling (ESM) family, to increase the accessibility of PLMs. By implementing optimizations such as FlashAttention and Partition-Attention, a novel technique designed to handle proteins of variable length, we achieved a 16-fold speedup in inference time and reduced memory usage by 3 to 14 times for long proteins. Additionally, 4-bit quantization applied to billion-parameter models led to a 2 to 3 times reduction in memory consumption with minimal performance loss in the missense variant effect prediction task. Training efficiency was also improved, with a 6-fold reduction in runtime achieved through activation checkpointing and the DeepSpeed Zero-Offload strategy. For fine-tuning, we employed parameter-efficient methods, enabling state-of-the-art predictions of protein properties and functions by training only the model head or a small fraction of adapter weights. For instance, we achieved a Spearman's correlation coefficient of 70% in melting point prediction and an 87% area under the precision-recall curve (AU-PRC) for transcription factor prediction. Our efficient ESM (ESME) implementation significantly lowers the barrier to using these powerful models, making them accessible to academic laboratories with limited computational resources. The ESME implementation is available on PyPI (pypi.org/project/esm-efficient) and GitHub (github.com/uci-cbcl/esm-efficient).

Bioinformatics
Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model

Yang Tan,Ruilin Wang,Banghao Wu,Liang Hong,Bingxin Zhou

2024-10-28

Abstract:Enzyme engineering enables the modification of wild-type proteins to meet industrial and research demands by enhancing catalytic activity, stability, binding affinities, and other properties. The emergence of deep learning methods for protein modeling has demonstrated superior results at lower costs compared to traditional approaches such as directed evolution and rational design. In mutation effect prediction, the key to pre-training deep learning models lies in accurately interpreting the complex relationships among protein sequence, structure, and function. This study introduces a retrieval-enhanced protein language model for comprehensive analysis of native properties from sequence and local structural interactions, as well as evolutionary properties from retrieved homologous sequences. The state-of-the-art performance of the proposed ProtREM is validated on over 2 million mutants across 217 assays from an open benchmark (ProteinGym). We also conducted post-hoc analyses of the model's ability to improve the stability and binding affinity of a VHH antibody. Additionally, we designed 10 new mutants on a DNA polymerase and conducted wet-lab experiments to evaluate their enhanced activity at higher temperatures. Both in silico and experimental evaluations confirmed that our method provides reliable predictions of mutation effects, offering an auxiliary tool for biologists aiming to evolve existing enzymes. The implementation is publicly available at <a class="link-external link-https" href="https://github.com/tyang816/ProtREM" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Quantitative Methods
Accurate prediction of functional effect of single amino acid variants with deep learning

Houssemeddine Derbel,Zhongming Zhao,Qian Liu

DOI: https://doi.org/10.1016/j.csbj.2023.11.017

IF: 6.155

2023-01-01

Computational and Structural Biotechnology Journal

Abstract:The assessment of functional effect of amino acid variants is a critical biological problem in proteomics for clinical medicine and protein engineering. Although natively occurring variants offer insights into deleterious variants, high-throughput deep mutational experiments enable comprehensive investigation of amino acid variants for a given protein. However, these mutational experiments are too expensive to dissect millions of variants on thousands of proteins. Thus, computational approaches have been proposed, but they heavily rely on hand-crafted evolutionary conservation, limiting their accuracy. Recent advancement in transformers provides a promising solution to precisely estimate the functional effects of protein variants on high-throughput experimental data. Here, we introduce a novel deep learning model, namely Rep2Mut-V2, which leverages learned representation from transformer models. Rep2Mut-V2 significantly enhances the prediction accuracy for 27 types of measurements of functional effects of protein variants. In the evaluation of 38 protein datasets with 118,933 single amino acid variants, Rep2Mut-V2 achieved an average Spearman's correlation coefficient of 0.7. This surpasses the performance of six state-of-the-art methods, including the recently released methods ESM, DeepSequence and EVE. Even with limited training data, Rep2Mut-V2 outperforms ESM and DeepSequence, showing its potential to extend high-throughput experimental analysis for more protein variants to reduce experimental cost. In conclusion, Rep2Mut-V2 provides accurate predictions of the functional effects of single amino acid variants of protein coding sequences. This tool can significantly aid in the interpretation of variants in human disease studies.

biochemistry & molecular biology
Design of the first highly potent and selective aminopeptidase N (EC 3.4.11.2) inhibitor

Huixiong Chen,B. Roques,M. Fournie-Zaluski

DOI: https://doi.org/10.1016/S0960-894X(99)00219-X

IF: 2.94

1999-06-07

Bioorganic & Medicinal Chemistry Letters

Abstract:
Enhancing missense variant pathogenicity prediction with protein language models using VariPred

Weining Lin,Jude Wells,Zeyuan Wang,Christine Orengo,Andrew C. R. Martin

DOI: https://doi.org/10.1038/s41598-024-51489-7

IF: 4.6

2024-04-09

Scientific Reports

Abstract:Computational approaches for predicting the pathogenicity of genetic variants have advanced in recent years. These methods enable researchers to determine the possible clinical impact of rare and novel variants. Historically these prediction methods used hand-crafted features based on structural, evolutionary, or physiochemical properties of the variant. In this study we propose a novel framework that leverages the power of pre-trained protein language models to predict variant pathogenicity. We show that our approach VariPred (Variant impact Predictor) outperforms current state-of-the-art methods by using an end-to-end model that only requires the protein sequence as input. Using one of the best-performing protein language models (ESM-1b), we establish a robust classifier that requires no calculation of structural features or multiple sequence alignments. We compare the performance of VariPred with other representative models including 3Cnet, Polyphen-2, REVEL, MetaLR, FATHMM and ESM variant. VariPred performs as well as, or in most cases better than these other predictors using six variant impact prediction benchmarks despite requiring only sequence data and no pre-processing of the data.

multidisciplinary sciences
Structure-informed protein language models are robust predictors for variant effects

Yuanfei Sun,Yang Shen

DOI: https://doi.org/10.1007/s00439-024-02695-w

2024-08-10

Human Genetics

Abstract:Emerging variant effect predictors, protein language models (pLMs) learn evolutionary distribution of functional sequences to capture fitness landscape. Considering that variant effects are manifested through biological contexts beyond sequence (such as structure), we first assess how much structure context is learned in sequence-only pLMs and affecting variant effect prediction. And we establish a need to inject into pLMs protein structural context purposely and controllably. We thus introduce a framework of structure-informed pLMs (SI-pLMs), by extending masked sequence denoising to cross-modality denoising for both sequence and structure. Numerical results over deep mutagenesis scanning benchmarks show that our SI-pLMs, even when using smaller models and less data, are robustly top performers against competing methods including other pLMs, which shows that introducing biological context can be more effective at capturing fitness landscape than simply using larger models or bigger data. Case studies reveal that, compared to sequence-only pLMs, SI-pLMs can be better at capturing fitness landscape because (a) learned embeddings of low/high-fitness sequences can be more separable and (b) learned amino-acid distributions of functionally and evolutionarily conserved residues can be of much lower entropy, thus much more conserved, than other residues. Our SI-pLMs are applicable to revising any sequence-only pLMs through model architecture and training objectives. They do not require structure data as model inputs for variant effect prediction and only use structures as context provider and model regularizer during training.

genetics & heredity

Viral transmission: Deadly contact

Expert-guided protein Language Models enable accurate and blazingly fast fitness prediction

Modeling the language of life – Deep Learning Protein Sequences

From a single sequence to evolutionary trajectories: protein language models capture the evolutionary potential of SARS-CoV-2 protein sequences

Assessing the role of evolutionary information for enhancing protein language model embeddings

Fine-tuning Protein Language Models with Deep Mutational Scanning improves Variant Effect Prediction

Alignment-free estimation of sequence conservation for identifying functional sites using protein sequence embeddings

Protein embeddings and deep learning predict binding residues for various ligand classes

Synthesis of inorganic polymers as glass precursors and for other uses: Pre‐ceramic block or graft copolymers as potential precursors to composite materials

Updated benchmarking of variant effect predictors using deep mutational scanning

TransEFVP: A Two-Stage Approach for the Prediction of Human Pathogenic Variants Based on Protein Sequence Embedding Fusion

Genome-wide prediction of disease variant effects with a deep protein language model

Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations

Enhancing predictions of protein stability changes induced by single mutations using MSA-based language models

Benchmarking text-integrated protein language model embeddings and embedding fusion on diverse downstream tasks

Efficient Inference, Training, and Fine-tuning of Protein Language Models

Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model

Accurate prediction of functional effect of single amino acid variants with deep learning

Design of the first highly potent and selective aminopeptidase N (EC 3.4.11.2) inhibitor

Enhancing missense variant pathogenicity prediction with protein language models using VariPred

Structure-informed protein language models are robust predictors for variant effects