Abstract:Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at <a class="link-external link-https" href="https://github.com/HazyResearch/hyena-dna" rel="external noopener nofollow">this https URL</a>.

DeepGene: An Efficient Foundation Model for Genomics based on Pan-genome Graph Transformer

dnaGrinder: a lightweight and high-capacity genomic foundation model

The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics

DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome

Benchmarking DNA Foundation Models for Genomic Sequence Classification

Enhancing Personalized Gene Expression Prediction From DNA Sequences Using Genomic Foundation Models

Deep Learning for Genomics: From Early Neural Nets to Modern Large Language Models

DeepGOA: Predicting Gene Ontology Annotations of Proteins Via Graph Convolutional Network

Genomic Interpreter: A Hierarchical Genomic Deep Neural Network with 1D Shifted Window Transformer

Accurate and General DNA Representations Emerge from Genome Foundation Models at Scale

GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences

SegmentNT: annotating the genome at single-nucleotide resolution with DNA foundation models

scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI

Coding genomes with gapped pattern graph convolutional network

Deciphering the Language of Protein-DNA Interactions: A Deep Learning Approach Combining Contextual Embeddings and Multi-Scale Sequence Modeling

DeepDNA: a Hybrid Convolutional and Recurrent Neural Network for Compressing Human Mitochondrial Genomes

scGREAT: Transformer-Based Deep-Language Model for Gene Regulatory Network Inference from Single-Cell Transcriptomics

GenoHoption: Bridging Gene Network Graphs and Single-Cell Foundation Models

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Genomics-FM: Universal Foundation Model for Versatile and Data-Efficient Functional Genomic Analysis