Abstract:Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. Such protein LMs cannot extrapolate to longer proteins and protein complexes well. They also fail to account for the underlying biological mechanisms carried out by biomolecular interactions and dynamics i.e., proteins often interact with other proteins, molecules, and pathways in complex biological systems. In this work, we propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built off selective structured state-space models, to learn high-quality universal protein representations at the amino acid token level using masked language modeling. We also introduce its graph-contextual variant, LC-PLM-G, which contextualizes protein-protein interaction (PPI) graphs for a second stage of training. LC-PLM demonstrates favorable neural scaling laws, better length extrapolation capability, and a 7% to 34% improvement on protein downstream tasks than Transformer-based ESM-2. LC-PLM-G further trained within the context of PPI graphs shows promising results on protein structure and function prediction tasks. Our study demonstrates the benefit of increasing the context size with computationally efficient LM architecture (e.g. structured state space models) in learning universal protein representations and incorporating molecular interaction context contained in biological graphs.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on the limitations of protein language models (pLMs) in dealing with long - sequence proteins and protein complexes. Specifically, most of the existing protein language models are based on the Transformer architecture, and these models use single - protein sequences with a relatively short context length during training. Consequently, they perform poorly when handling longer protein sequences and protein complexes, and it is difficult for them to capture the complex biomolecular interactions and kinetic mechanisms among proteins. Moreover, these models provide insufficient support for tasks that require long - range dependencies, such as genome interactions, protein - protein interactions (PPI), protein function prediction, and 3D structure prediction of long proteins and protein complexes. To address these issues, the author proposes a new protein language model based on Selective Structured State - Space Models (SSSMs), namely LC - PLM (Long - Context Protein Language Model). This model aims to learn high - quality general protein representations through an improved bidirectional Mamba architecture (BiMamba - S), thereby enhancing the ability to handle long sequences and model biomolecular interactions. In addition, the author also introduces a graph - context variant LC - PLM - G, which further enhances the model's ability to capture the context of biomolecular interactions through a second - stage training in the protein - protein interaction graph. Overall, the goal of this paper is to develop a protein language model that can effectively handle long - sequence proteins and their complexes and better capture biomolecular interaction information, so as to improve performance in related downstream tasks such as protein structure prediction and function prediction.

Long-context Protein Language Model

ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training

Learning immune receptor representations with protein language models

S-PLM: Structure-aware Protein Language Model via Contrastive Learning between Sequence and Structure

InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders

PLM-interact: extending protein language models to predict protein-protein interactions

MULAN: Multimodal Protein Language Model for Sequence and Structure Encoding

ProLLaMA: A Protein Language Model for Multi-Task Protein Language Processing

Structure-Infused Protein Language Models

Protein language models meet reduced amino acid alphabets

Training Compute-Optimal Protein Language Models

DPLM-2: A Multimodal Diffusion Protein Language Model

Efficient Inference, Training, and Fine-tuning of Protein Language Models

Protein language models learn evolutionary statistics of interacting sequence motifs

InstructPLM: Aligning Protein Language Models to Follow Protein Structure Instructions

ProLLM: Protein Chain-of-Thoughts Enhanced LLM for Protein-Protein Interaction Prediction

Design Proteins Using Large Language Models: Enhancements and Comparative Analyses

Modeling Protein Using Large-scale Pretrain Language Model

Protein Language Models and Structure Prediction: Connection and Progression

Does protein pretrained language model facilitate the prediction of protein–ligand interaction?

Diffusion Language Models Are Versatile Protein Learners