Abstract:Background: In the evolving landscape of microbiology and microbiome analysis, the integration of machine learning is crucial for understanding complex microbial interactions, and predicting and recognizing novel functionalities within extensive datasets. However, the effectiveness of these methods in microbiology faces challenges due to the complex and heterogeneous nature of microbial data, further complicated by low signal-to-noise ratios, context-dependency, and a significant shortage of appropriately labeled datasets. This study introduces the ProkBERT model family, a collection of large language models, designed for genomic tasks. It provides a generalizable sequence representation for nucleotide sequences, learned from unlabeled genome data. This approach helps overcome the above-mentioned limitations in the field, thereby improving our understanding of microbial ecosystems and their impact on health and disease. Methods: ProkBERT models are based on transfer learning and self-supervised methodologies, enabling them to use the abundant yet complex microbial data effectively. The introduction of the novel Local Context-Aware (LCA) tokenization technique marks a significant advancement, allowing ProkBERT to overcome the contextual limitations of traditional transformer models. This methodology not only retains rich local context but also demonstrates remarkable adaptability across various bioinformatics tasks. Results: In practical applications such as promoter prediction and phage identification, the ProkBERT models show superior performance. For promoter prediction tasks, the top-performing model achieved a Matthews Correlation Coefficient (MCC) of 0.74 for E. coli and 0.62 in mixed-species contexts. In phage identification, ProkBERT models consistently outperformed established tools like VirSorter2 and DeepVirFinder, achieving an MCC of 0.85. These results underscore the models' exceptional accuracy and generalizability in both supervised and unsupervised tasks. Conclusions: The ProkBERT model family is a compact yet powerful tool in the field of microbiology and bioinformatics. Its capacity for rapid, accurate analyses and its adaptability across a spectrum of tasks marks a significant advancement in machine learning applications in microbiology. The models are available on GitHub (https://github.com/nbrg-ppcu/prokbert) and HuggingFace (https://huggingface.co/nerualbioinfo) providing an accessible tool for the community.

ChromBERT: Uncovering Chromatin State Motifs in the Human Genome Using a BERT-based Approach

Chromatin Potential Identified by Shared Single Cell Profiling of RNA and Chromatin

Improving language model of human genome for DNA–protein binding prediction based on task-specific pre-training

BertSNR: an interpretable deep learning framework for single-nucleotide resolution identification of transcription factor binding sites based on DNA language model

IChrom-Deep: An Attention-Based Deep Learning Model for Identifying Chromatin Interactions

The role of chromatin state in intron retention: a case study in leveraging large scale deep learning models

Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin

ChromDL: A Next-Generation Regulatory DNA Classifier

Decoding chromatin states by proteomic profiling of nucleosome readers

BERT-TFBS: a novel BERT-based model for predicting transcription factor binding sites by transfer learning

Toward Understanding BERT-Like Pre-Training for DNA Foundation Models

stBERT: A Pretrained Model for Spatial Domain Identification of Spatial Transcriptomics

Predictive Recognition of DNA-binding Proteins Based on Pre-trained Language Model BERT

A transformer architecture based on BERT and 2D convolutional neural network to identify DNA enhancers from sequence information

Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types

EnhancerBD identifing sequence feature

A generalizable framework to comprehensively predict epigenome, chromatin organization, and transcriptome

Deciphering 3'UTR Mediated Gene Regulation Using Interpretable Deep Representation Learning

Chromatin Accessibility Prediction Via Convolutional Long Short-Term Memory Networks with K-Mer Embedding.

ProkBERT family: genomic language models for microbiome applications

DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome