Are Genomic Language Models All You Need? Exploring Genomic Language Models on Protein Downstream Tasks

Sam Boshar,Evan Trop,Bernardo P de Almeida,Liviu Copoiu,Thomas Pierrot

DOI: https://doi.org/10.1101/2024.05.20.594989

2024-05-22

Abstract:Motivation: Large language models, trained on enormous corpora of biological sequences, are state-of-the-art for downstream genomic and proteomic tasks. Since the genome contains the information to encode all proteins, genomic language models (gLMs) hold the potential to make downstream predictions not only about DNA sequences, but also about proteins. However, the performance of gLMs on protein tasks remains unknown, due to few tasks pairing proteins with the coding DNA sequences (CDS) that can be processed by gLMs. Results: In this work, we curated five such datasets and used them to evaluate the performance of gLMs and proteomic language models (pLMs). We show that gLMs are competitive and even outperform their pLMs counterparts on some tasks. The best performance was achieved using the retrieved CDS compared to sampling strategies. We found that training a joint genomic-proteomic model outperforms each individual approach, showing that they capture different but complementary sequence representations, as we demonstrate through model interpretation of their embeddings. Lastly, we explored different genomic tokenization schemes to improve downstream protein performance. We trained a new Nucleotide Transformer (50M) foundation model with 3mer tokenization that outperforms its 6mer counterpart on protein tasks while maintaining performance on genomics tasks. The application of gLMs to proteomics offers the potential to leverage rich CDS data, and in the spirit of the central dogma, the possibility of a unified and synergistic approach to genomics and proteomics. Availability and implementation: We make our inference code, 3mer pre-trained model weights and datasets available.

Bioinformatics

What problem does this paper attempt to address?

This paper explores the performance of genomic language models (gLMs) on protein-related tasks. The researchers found that although gLMs are primarily trained on non-coding regions, they compete with or even outperform protein language models (pLMs) on some protein tasks. By comparing different datasets and strategies, they demonstrated the importance of obtaining genuine coding DNA sequences (CDS) for gLM performance. In addition, they trained a 3-nucleotide (3mer) gLM model to improve tasks sensitive to amino acid-level variations, but with limited success. The paper also proposes that combining gLMs and pLMs can leverage their respective strengths, resulting in joint models that achieve or surpass the current state-of-the-art in four tasks and set a new standard in protein stability prediction. The research findings indicate that gLMs are able to capture protein information but may be inferior to pLMs in capturing fine amino acid structures or single-point mutations. Overall, this paper aims to evaluate the potential of gLMs as a unified approach for handling genomics and proteomics tasks.