Abstract:Background: In the evolving landscape of microbiology and microbiome analysis, the integration of machine learning is crucial for understanding complex microbial interactions, and predicting and recognizing novel functionalities within extensive datasets. However, the effectiveness of these methods in microbiology faces challenges due to the complex and heterogeneous nature of microbial data, further complicated by low signal-to-noise ratios, context-dependency, and a significant shortage of appropriately labeled datasets. This study introduces the ProkBERT model family, a collection of large language models, designed for genomic tasks. It provides a generalizable sequence representation for nucleotide sequences, learned from unlabeled genome data. This approach helps overcome the above-mentioned limitations in the field, thereby improving our understanding of microbial ecosystems and their impact on health and disease. Methods: ProkBERT models are based on transfer learning and self-supervised methodologies, enabling them to use the abundant yet complex microbial data effectively. The introduction of the novel Local Context-Aware (LCA) tokenization technique marks a significant advancement, allowing ProkBERT to overcome the contextual limitations of traditional transformer models. This methodology not only retains rich local context but also demonstrates remarkable adaptability across various bioinformatics tasks. Results: In practical applications such as promoter prediction and phage identification, the ProkBERT models show superior performance. For promoter prediction tasks, the top-performing model achieved a Matthews Correlation Coefficient (MCC) of 0.74 for E. coli and 0.62 in mixed-species contexts. In phage identification, ProkBERT models consistently outperformed established tools like VirSorter2 and DeepVirFinder, achieving an MCC of 0.85. These results underscore the models' exceptional accuracy and generalizability in both supervised and unsupervised tasks. Conclusions: The ProkBERT model family is a compact yet powerful tool in the field of microbiology and bioinformatics. Its capacity for rapid, accurate analyses and its adaptability across a spectrum of tasks marks a significant advancement in machine learning applications in microbiology. The models are available on GitHub (https://github.com/nbrg-ppcu/prokbert) and HuggingFace (https://huggingface.co/nerualbioinfo) providing an accessible tool for the community.

Leveraging Large Language Models for Metagenomic Analysis

ProkBERT family: genomic language models for microbiome applications

genomicBERT and data-free deep-learning model evaluation

Leveraging pre-trained language models for mining microbiome-disease relationships

Learning a deep language model for microbiomes: the power of large scale unlabeled microbiome data

Analyzing Large Microbiome Datasets Using Machine Learning and Big Data

FGBERT: Function-Driven Pre-trained Gene Language Model for Metagenomics

An Analysis on Large Language Models in Healthcare: A Case Study of BioBERT

Recent advances in deep learning and language models for studying the microbiome

Harnessing Machine Learning for Metagenomics: Discovering the Invisible Microbial World

Large language models in bioinformatics: applications and perspectives

Microbial general model: Leveraging large language model for contextualized microbiome analysis

scReader: Prompting Large Language Models to Interpret scRNA-seq Data

Deciphering enzymatic potential in metagenomic reads through DNA language model

Advancing Plant Metabolic Research By Using Large Language Models To Expand Databases And Extract Labelled Data

Large-scale Machine Learning for Metagenomics Sequence Classification

Improving Taxonomic Classification with Feature Space Balancing

GeneGPT: augmenting large language models with domain tools for improved access to biomedical information

ProkBERT PhaStyle: Accurate Phage Lifestyle Prediction with Pretrained Genomic Language Models

ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing