Abstract:Introduction:: More recent self-supervised deep language models, such as Bidirectional Encoder Representations from Transformers (BERT), have performed the best on some language tasks by contextualizing word embeddings for a better dynamic representation. Their proteinspecific versions, such as ProtBERT, generated dynamic protein sequence embeddings, which resulted in better performance for several bioinformatics tasks. Besides, a number of different protein post-translational modifications are prominent in cellular tasks such as development and differentiation. The current biological experiments can detect these modifications, but within a longer duration and with a significant cost. Methods:: In this paper, to comprehend the accompanying biological processes concisely and more rapidly, we propose DEEPPTM to predict protein post-translational modification (PTM) sites from protein sequences more efficiently. Different than the current methods, DEEPPTM enhances the modification prediction performance by integrating specialized ProtBERT-based protein embeddings with attention-based vision transformers (ViT), and reveals the associations between different modification types and protein sequence content. Additionally, it can infer several different modifications over different species. Results:: Human and mouse ROC AUCs for predicting Succinylation modifications were 0.988 and 0.965 respectively, once 10-fold cross-validation is applied. Similarly, we have obtained 0.982, 0.955, and 0.953 ROC AUC scores on inferring ubiquitination, crotonylation, and glycation sites, respectively. According to detailed computational experiments, DEEPPTM lessens the time spent in laboratory experiments while outperforming the competing methods as well as baselines on inferring all 4 modification sites. In our case, attention-based deep learning methods such as vision transformers look more favorable to learning from ProtBERT features than more traditional deep learning and machine learning techniques. Conclusion:: Additionally, the protein-specific ProtBERT model is more effective than the original BERT embeddings for PTM prediction tasks. Our code and datasets can be found at https://github.com/seferlab/deepptm.

FAD-BERT: Improved prediction of FAD binding sites using pre-training of deep bidirectional transformers

FEDBFPT: an Efficient Federated Learning Framework for BERT Further Pre-Training

BERT-TFBS: a novel BERT-based model for predicting transcription factor binding sites by transfer learning

A novel fusion technology utilizing complex network and sequence information for FAD-binding site identification

Improving language model of human genome for DNA–protein binding prediction based on task-specific pre-training

Prediction of FMN Binding Sites in Electron Transport Chains Based on 2-D CNN and PSSM Profiles

PLAPT: Protein-Ligand Binding Affinity Prediction Using Pretrained Transformers

BBATProt: A Framework Predicting Biological Function with Enhanced Feature Extraction via Explainable Deep Learning

Predictive Recognition of DNA-binding Proteins Based on Pre-trained Language Model BERT

Predicting Transcription Factor Binding Sites using Transformer based Capsule Network

Predicting ATP binding sites in protein sequences using Deep Learning and Natural Language Processing

MREDTA: A BERT and transformer-based molecular representation encoder for predicting drug-target binding affinity

DDA-BERT: leveraging transformer architecture pre-training for data-dependent acquisition mass spectrometry-based proteomics

Advancing Drug-Target Interaction prediction with BERT and subsequence embedding

BTFBS: binding-prediction of bacterial transcription factors and binding sites based on deep learning

BERT-Kcr: prediction of lysine crotonylation sites by a transfer learning method with pre-trained BERT models

G-K BertDTA: A graph representation learning and semantic embedding-based framework for drug-target affinity prediction

Predicting Transcription Factor Binding Sites with Deep Learning

Predicting Binding from Screening Assays with Transformer Network Embeddings

DeepPTM: Protein Post-translational Modification Prediction from Protein Sequences by Combining Deep Protein Language Model with Vision Transformers

To Improve Prediction of Binding Residues with DNA, RNA, Carbohydrate, and Peptide Via Multi-Task Deep Neural Networks