Abstract:Protein language models have shown great promise in predicting protein structure, function, and the effects of missense variants on protein fitness. However, their use has been limited by the substantial computational resources required. In this work, we focus on improving the computational efficiency of protein language models (PLMs), specifically the Evolutionary Scale Modeling (ESM) family, to increase the accessibility of PLMs. By implementing optimizations such as FlashAttention and Partition-Attention, a novel technique designed to handle proteins of variable length, we achieved a 16-fold speedup in inference time and reduced memory usage by 3 to 14 times for long proteins. Additionally, 4-bit quantization applied to billion-parameter models led to a 2 to 3 times reduction in memory consumption with minimal performance loss in the missense variant effect prediction task. Training efficiency was also improved, with a 6-fold reduction in runtime achieved through activation checkpointing and the DeepSpeed Zero-Offload strategy. For fine-tuning, we employed parameter-efficient methods, enabling state-of-the-art predictions of protein properties and functions by training only the model head or a small fraction of adapter weights. For instance, we achieved a Spearman's correlation coefficient of 70% in melting point prediction and an 87% area under the precision-recall curve (AU-PRC) for transcription factor prediction. Our efficient ESM (ESME) implementation significantly lowers the barrier to using these powerful models, making them accessible to academic laboratories with limited computational resources. The ESME implementation is available on PyPI (pypi.org/project/esm-efficient) and GitHub (github.com/uci-cbcl/esm-efficient).

Fine-tuning of conditional Transformers for the generation of functionally characterized enzymes

Conditional language models enable the efficient design of proficient enzymes

Conditional Enzyme Generation Using Protein Language Models with Adapters

Active Finetuning Protein Language Model: A Budget-Friendly Method for Directed Evolution

Identification of Enzymatic Active Sites with Unsupervised Language Modeling

Predicting Enzymatic Reactions with a Molecular Transformer

Knowledge-aware Reinforced Language Models for Protein Directed Evolution

InstructPLM: Aligning Protein Language Models to Follow Protein Structure Instructions

Fine-tuning protein language models boosts predictions across diverse tasks

Integrating Genetic Algorithms and Language Models for Enhanced Enzyme Design

De novo design of triosephosphate isomerases using generative language models

Identifying promising sequences for protein engineering using a deep transformer protein language model

Generative Enzyme Design Guided by Functionally Important Sites and Small-Molecule Substrates

Efficient Inference, Training, and Fine-tuning of Protein Language Models

Adapt-cMolGPT: A Conditional Generative Pre-Trained Transformer with Adapter-Based Fine-Tuning for Target-Specific Molecular Generation

Exploring evolution-aware & -free protein language models as protein function predictors

Evaluation of Reinforcement Learning in Transformer-based Molecular Design

Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding

Biophysics-based protein language models for protein engineering

GOProFormer: A Multi-modal Transformer Method for Gene Ontology Protein Function Prediction

Llamol: a dynamic multi-conditional generative transformer for de novo molecular design