Abstract:Multi-modality pre-training paradigm that aligns protein sequences and biological descriptions has learned general protein representations and achieved promising performance in various downstream applications. However, these works were still unable to replicate the extraordinary success of language-supervised visual foundation models due to the ineffective usage of aligned protein-text paired data and the lack of an effective function-informed pre-training paradigm. To address these issues, this paper curates a large-scale protein-text paired dataset called ProtAnno with a property-driven sampling strategy, and introduces a novel function-informed protein pre-training paradigm. Specifically, the sampling strategy determines selecting probability based on the sample confidence and property coverage, balancing the data quality and data quantity in face of large-scale noisy data. Furthermore, motivated by significance of the protein specific functional mechanism, the proposed paradigm explicitly model protein static and dynamic functional segments by two segment-wise pre-training objectives, injecting fine-grained information in a function-informed manner. Leveraging all these innovations, we develop ProtCLIP, a multi-modality foundation model that comprehensively represents function-aware protein embeddings. On 22 different protein benchmarks within 5 types, including protein functionality classification, mutation effect prediction, cross-modal transformation, semantic similarity inference and protein-protein interaction prediction, our ProtCLIP consistently achieves SOTA performance, with remarkable improvements of 75% on average in five cross-modal transformation benchmarks, 59.9% in GO-CC and 39.7% in GO-BP protein function prediction. The experimental results verify the extraordinary potential of ProtCLIP serving as the protein multi-modality foundation model.

Multimodal pretraining for unsupervised protein representation learning

ProteinAligner: A Multi-modal Pretraining Framework for Protein Foundation Models

Protein Representation Learning by Geometric Structure Pretraining

Protein Representation Learning by Capturing Protein Sequence-Structure-Function Relationship

Multi-level protein pre-training with Vabs-Net

Multimodal Pre-Training Model for Sequence-based Prediction of Protein-Protein Interaction

Pre-Training Protein Bi-level Representation Through Span Mask Strategy On 3D Protein Chains

Pre-Training of Deep Bidirectional Protein Sequence Representations With Structural Information

A Systematic Study of Joint Representation Learning on Protein Sequences and Structures

Pre-training Sequence, Structure, and Surface Features for Comprehensive Protein Representation Learning

SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation

SPRoBERTa: protein embedding learning with local fragment modeling

Geometric Self-Supervised Pretraining on 3D Protein Structures using Subgraphs

Modeling Protein Using Large-scale Pretrain Language Model

ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts

ProteiNexus: Illuminating Protein Pathways through Structural Pre-training

ProtCLIP: Function-Informed Protein Multi-Modal Learning

Progressive Multi-Modality Learning for Inverse Protein Folding

Contrastive Representation Learning for 3D Protein Structures