Abstract:Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at <a class="link-external link-https" href="https://github.com/HazyResearch/hyena-dna" rel="external noopener nofollow">this https URL</a>.

DeepDNA: a Hybrid Convolutional and Recurrent Neural Network for Compressing Human Mitochondrial Genomes

Human Mitochondrial Genome Compression Using Machine Learning Techniques

Dna-Compact: Dna Compression Based on A Pattern-Aware Contextual Modeling Technique

GenCoder: A Novel Convolutional Neural Network based Autoencoder for Genomic Sequence Data Compression

DeepGene: An Efficient Foundation Model for Genomics based on Pan-genome Graph Transformer

Genetic Sequence compression using Machine Learning and Arithmetic Encoding Decoding Techniques

Deep Joint Source-Channel Coding for DNA Image Storage: A Novel Approach with Enhanced Error Resilience and Biological Constraint Optimization

Implicit Neural Multiple Description for DNA-based data storage

A Novel Compression Algorithm for High-Throughput DNA Sequence Based on Huffman Coding Method

GeneFormer: Learned Gene Compression using Transformer-based Context Modeling

Deep DNA Storage: Scalable and Robust DNA Storage via Coding Theory and Deep Learning

Coding genomes with gapped pattern graph convolutional network

DanQ: a hybrid convolutional and recurrent deep neural network for quantifying the function of DNA sequences

GPU-accelerated adaptive compression framework for genomics data

The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics

A Novel Compression Tool For Efficient Storage Of Genome Resequencing Data

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Single-Read Reconstruction for DNA Data Storage Using Transformers

FECDO-Flexible and Efficient Coding for DNA Odyssey

DeepFormer: a hybrid network based on convolutional neural network and flow-attention mechanism for identifying the function of DNA sequences

CoGI: Towards Compressing Genomes As an Image.