Abstract:The spatial organization of the genome within the nucleus is partially determined by its interactions with distinct nuclear subcompartments, such as the nuclear lamina and nuclear speckles, which play key roles in gene regulation during development. However, whether these genome-nuclear subcompartment interactions are encoded in the underlying DNA sequence remains poorly understood. The mechanisms for gene regulation are primarily encoded in noncoding DNA sequences, but deciphering how these sequence features control gene expression remains a significant challenge in genomics. Here, we present Nucleotide GPT, a transformer-based model that predicts genomic associations with spatially distinct, physical nuclear subcompartments from DNA sequence alone. Pre-trained on a diverse set of multi-species genomes, we demonstrate Nucleotide GPT's genomic understanding through evaluation on diverse prediction tasks, including histone modifications, promoter detection, and transcription factor binding sites. When fine-tuned to predict genome interactions with two separate nuclear subcompartments - the lamina of the inner nuclear membrane and nuclear speckles that lie more interior - Nucleotide GPT achieves an average accuracy of 73.6% for lamina-associated domains (LADs) and 79.4% accuracy for speckle-associated domains (SPADs), averaged across three cortical development cell types. Analysis of the model's learned representations through Uniform Manifold Approximation and Projection (UMAP) reveals that Nucleotide GPT develops internal embeddings that effectively distinguish LADs from inter-LADs, with predicted probabilities closely corresponding to experimentally determined LAD classifications. When examining these representations in the context of cell type-invariant constitutive LADs (cLADs) compared to cell type-specific LADs, the model assigns lower confidence scores to cell type-specific LADs compared to cLADs that are conserved across neuronal differentiation, suggesting sequence features may play a stronger role in maintaining cLAD associations. Examination of the model's attention patterns at correctly classified regions suggests that specific sequence elements govern model decision making about nuclear subcompartment associations. Our results demonstrate the utility of transformer architectures for studying three-dimensional (3D) genome organization and substantiate a role for DNA sequence in determining nuclear subcompartment associations.

EpiGePT: a Pretrained Transformer model for epigenomics

Capturing Large Genomic Contexts for Accurately Predicting Enhancer-Promoter Interactions

EPI-Trans: an effective transformer-based deep learning model for enhancer promoter interaction prediction

TF-EPI: an interpretable enhancer-promoter interaction detection method based on Transformer

Prediction of enhancer-promoter interactions via natural language processing

EPI-Mind: Identifying Enhancer–Promoter Interactions Based on Transformer Mechanism

A generalizable framework to comprehensively predict epigenome, chromatin organization, and transcriptome

Epigenetics is all you need: A Transformer to decode chromatin structural compartments from the epigenome

L2G: Repurposing Language Models for Genomics Tasks

Nucleotide GPT: Sequence-Based Deep Learning Prediction of Nuclear Subcompartment-Associated Genome Architecture

Exploiting Epigenomic and Sequence-Based Features for Predicting Enhancer-Promoter Interactions

EpiTEAmDNA: Sequence feature representation via transfer learning and ensemble learning for identifying multiple DNA epigenetic modification types across species

Enhancing Personalized Gene Expression Prediction From DNA Sequences Using Genomic Foundation Models

Integrating Prior Knowledge Using Transformer for Gene Regulatory Network Inference

Predicting cell type-specific epigenomic profiles accounting for distal genetic effects

EPInformer: A Scalable Deep Learning Framework for Gene Expression Prediction by Integrating Promoter-enhancer Sequences with Multimodal Epigenomic Data

TRAPT: A multi-stage fused deep learning framework for transcriptional regulators prediction via integrating large-scale epigenomic data

Identifying enhancer-promoter interactions with neural network based on pre-trained DNA vectors and attention mechanism

MuLan-Methyl-multiple transformer-based language models for accurate DNA methylation prediction

scKEPLM: Knowledge enhanced large-scale pre-trained language model for single-cell transcriptomics

MethylGPT: a foundation model for the DNA methylome