Abstract:Protein Language Models (pLMs) have revolutionized the computational modeling of protein systems, building numerical embeddings that are centered around structural features. To enhance the breadth of biochemically relevant properties available in protein embeddings, we engineered the Annotation Vocabulary, a transformer readable language of protein properties defined by structured ontologies. We trained Annotation Transformers (AT) from the ground up to recover masked protein property inputs without reference to amino acid sequences, building a new numerical feature space on protein descriptions alone. We leverage AT representations in various model architectures, for both protein representation and generation. To showcase the merit of Annotation Vocabulary integration, we performed 515 diverse downstream experiments. Using a novel loss function and only $3 in commercial compute, our premier representation model CAMP produces state-of-the-art embeddings for five out of 15 common datasets with competitive performance on the rest; highlighting the computational efficiency of latent space curation with Annotation Vocabulary. To standardize the comparison of de novo generated protein sequences, we suggest a new sequence alignment-based score that is more flexible and biologically relevant than traditional language modeling metrics. Our generative model, GSM, produces high alignment scores from annotation-only prompts with a BERT-like generation scheme. Of particular note, many GSM hallucinations return statistically significant BLAST hits, where enrichment analysis shows properties matching the annotation prompt - even when the ground truth has low sequence identity to the entire training set. Overall, the Annotation Vocabulary toolbox presents a promising pathway to replace traditional tokens with members of ontologies and knowledge graphs, enhancing transformer models in specific domains. The concise, accurate, and efficient descriptions of proteins by the Annotation Vocabulary offers a novel way to build numerical representations of proteins for protein annotation and design.

GOProFormer: A Multi-modal Transformer Method for Gene Ontology Protein Function Prediction

ProtGO: A Transformer based Fusion Model for accurately predicting Gene Ontology (GO) Terms from full scale Protein Sequences

DeepGOA: Predicting Gene Ontology Annotations of Proteins Via Graph Convolutional Network

TransformerGO: predicting protein-protein interactions by modelling the attention between sets of gene ontology terms

PROTGOAT : Improved automated protein function predictions using Protein Language Models

Protein Function Prediction With Functional and Topological Knowledge of Gene Ontology

Improving protein function prediction by learning and integrating representations of protein sequences and function labels

MultiPredGO: Deep Multi-Modal Protein Function Prediction by Amalgamating Protein Structure, Sequence, and Interaction Information

OntoProtein: Protein Pretraining With Gene Ontology Embedding

DEEPGONET: Multi-label Prediction of GO Annotation for Protein from Sequence Using Cascaded Convolutional and Recurrent Network

A Deep Learning Framework for Gene Ontology Annotations with Sequence- and Network-Based Information

Transformer-based deep learning for predicting protein properties in the life sciences

Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers

DeepGATGO: A Hierarchical Pretraining-Based Graph-Attention Model for Automatic Protein Function Prediction

Protein function prediction with gene ontology: from traditional to deep learning models

Transformer Neural Networks Attending to Both Sequence and Structure for Protein Prediction Tasks

Partial order relation–based gene ontology embedding improves protein function prediction

DeepGO: Predicting protein functions from sequence and interactions using a deep ontology-aware classifier

TALE: Transformer-based protein function Annotation with joint sequence–Label Embedding

Combining protein sequences and structures with transformers and equivariant graph neural networks to predict protein function

Annotation Vocabulary (Might Be) All You Need