Abstract:Proteomics has been revolutionized by large protein language models (PLMs), which learn unsupervised representations from large corpora of sequences. These models are typically fine-tuned in a supervised setting to adapt the model to specific downstream tasks. However, the computational and memory footprint of fine-tuning (FT) large PLMs presents a barrier for many research groups with limited computational resources. Natural language processing has seen a similar explosion in the size of models, where these challenges have been addressed by methods for parameter-efficient fine-tuning (PEFT). In this work, we introduce this paradigm to proteomics through leveraging the parameter-efficient method LoRA and training new models for two important tasks: predicting protein-protein interactions (PPIs) and predicting the symmetry of homooligomer quaternary structures. We show that these approaches are competitive with traditional FT while requiring reduced memory and substantially fewer parameters. We additionally show that for the PPI prediction task, training only the classification head also remains competitive with full FT, using five orders of magnitude fewer parameters, and that each of these methods outperform state-of-the-art PPI prediction methods with substantially reduced compute. We further perform a comprehensive evaluation of the hyperparameter space, demonstrate that PEFT of PLMs is robust to variations in these hyperparameters, and elucidate where best practices for PEFT in proteomics differ from those in natural language processing. All our model adaptation and evaluation code is available open-source at https://github.com/microsoft/peft_proteomics. Thus, we provide a blueprint to democratize the power of PLM adaptation to groups with limited computational resources.

Parameter-efficient fine-tuning on large protein language models improves signal peptide prediction

Democratizing protein language models with parameter-efficient fine-tuning

Fine-tuning protein language models boosts predictions across diverse tasks

Efficient Inference, Training, and Fine-tuning of Protein Language Models

Simple, Efficient and Scalable Structure-aware Adapter Boosts Protein Language Models

Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

Parameter-Efficient Fine-Tuning of State Space Models

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

Parameter-Efficient Fine-Tuning Enhances Adaptation of Single Cell Large Language Model for Cell Type Identification

Does protein pretrained language model facilitate the prediction of protein–ligand interaction?

SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models

Unbiased organism-agnostic and highly sensitive signal peptide predictor with deep protein language model

Interpretable improving prediction performance of general protein language model by domain-adaptive pretraining on DNA-binding protein

Protein-peptide binding residue prediction based on protein language models and cross-attention mechanism

SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models

ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation

LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

Assessing parameter efficient methods for pre-trained language model in annotating scRNA-seq data

Large language models capsule: A research analysis of In-Context Learning (ICL) and Parameter-Efficient Fine-Tuning (PEFT) methods

Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models