Abstract:Background: In the evolving landscape of microbiology and microbiome analysis, the integration of machine learning is crucial for understanding complex microbial interactions, and predicting and recognizing novel functionalities within extensive datasets. However, the effectiveness of these methods in microbiology faces challenges due to the complex and heterogeneous nature of microbial data, further complicated by low signal-to-noise ratios, context-dependency, and a significant shortage of appropriately labeled datasets. This study introduces the ProkBERT model family, a collection of large language models, designed for genomic tasks. It provides a generalizable sequence representation for nucleotide sequences, learned from unlabeled genome data. This approach helps overcome the above-mentioned limitations in the field, thereby improving our understanding of microbial ecosystems and their impact on health and disease. Methods: ProkBERT models are based on transfer learning and self-supervised methodologies, enabling them to use the abundant yet complex microbial data effectively. The introduction of the novel Local Context-Aware (LCA) tokenization technique marks a significant advancement, allowing ProkBERT to overcome the contextual limitations of traditional transformer models. This methodology not only retains rich local context but also demonstrates remarkable adaptability across various bioinformatics tasks. Results: In practical applications such as promoter prediction and phage identification, the ProkBERT models show superior performance. For promoter prediction tasks, the top-performing model achieved a Matthews Correlation Coefficient (MCC) of 0.74 for E. coli and 0.62 in mixed-species contexts. In phage identification, ProkBERT models consistently outperformed established tools like VirSorter2 and DeepVirFinder, achieving an MCC of 0.85. These results underscore the models' exceptional accuracy and generalizability in both supervised and unsupervised tasks. Conclusions: The ProkBERT model family is a compact yet powerful tool in the field of microbiology and bioinformatics. Its capacity for rapid, accurate analyses and its adaptability across a spectrum of tasks marks a significant advancement in machine learning applications in microbiology. The models are available on GitHub (https://github.com/nbrg-ppcu/prokbert) and HuggingFace (https://huggingface.co/nerualbioinfo) providing an accessible tool for the community.

Fine-tuning BERT models to extract transcriptional regulatory interactions of bacteria from biomedical literature

Automatic extraction of transcriptional regulatory interactions of bacteria from biomedical literature using a BERT-based approach

Leveraging pre-trained language models for mining microbiome-disease relationships

Integrating natural language processing and genome analysis enables accurate bacterial phenotype prediction

An Optimized Method for Reconstruction of Transcriptional Regulatory Networks in Bacteria Using ChIP-exo and RNA-seq Datasets

Fine-tuning large neural language models for biomedical natural language processing

From pre-training to fine-tuning: An in-depth analysis of Large Language Models in the biomedical domain

BactInt: A domain driven transfer learning approach for extracting inter-bacterial associations from biomedical text

Evaluation of GPT and BERT-based models on identifying proteinprotein interactions in biomedical text

Evaluating Biomedical BERT Models for Vocabulary Alignment at Scale in the UMLS Metathesaurus

A span-based joint model for extracting entities and relations of bacteria biotopes

Predicting bacterial fitness in Mycobacterium tuberculosis with transcriptional regulatory network-informed interpretable machine learning

Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT

Fine-Tuning Large Language Models for Scientific Text Classification: A Comparative Study

A Comprehensive Evaluation of Large Language Models in Mining Gene Interactions and Pathway Knowledge

BioBERT and Similar Approaches for Relation Extraction

A comprehensive evaluation of large language models in mining gene relations and pathway knowledge

Data-driven modeling of E. coli transcriptional regulation

Ensemble pretrained language models to extract biomedical knowledge from literature

ProkBERT family: genomic language models for microbiome applications