Abstract:Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at <a class="link-external link-https" href="https://github.com/HazyResearch/hyena-dna" rel="external noopener nofollow">this https URL</a>.

Accurate and General DNA Representations Emerge from Genome Foundation Models at Scale

GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences

dnaGrinder: a lightweight and high-capacity genomic foundation model

Codon language embeddings provide strong signals for use in protein engineering

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Scaling Dense Representations for Single Cell with Transcriptome-Scale Context

Benchmarking DNA Foundation Models for Genomic Sequence Classification

The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics

Sequence modeling and design from molecular to genome scale with Evo

Deciphering the Language of Protein-DNA Interactions: A Deep Learning Approach Combining Contextual Embeddings and Multi-Scale Sequence Modeling

DeepGene: An Efficient Foundation Model for Genomics based on Pan-genome Graph Transformer

DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome

BEND: Benchmarking DNA Language Models on biologically meaningful tasks

DNAHLM -- DNA sequence and Human Language mixed large language Model

Beyond DNA: ML-Empowered Nanopore Base-Calling of 12-Letter Genetic Alphabets

GENA-Web - GENomic Annotations Web Inference using DNA language models

Understanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precision

ProGen2: Exploring the Boundaries of Protein Language Models

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Generative Language Models on Nucleotide Sequences of Human Genes