Abstract:Background: Decoding human genomic sequences requires comprehensive analysis of DNA sequence functionality. Through computational and experimental approaches, researchers have studied the genotype-phenotype relationship and generate important datasets that help unravel complicated genetic blueprints. Thus, the recently developed artificial intelligence methods can be used to interpret the functions of those DNA sequences. Methods: This study explores the use of deep learning, particularly pre-trained genomic models like DNA_bert_6 and human_gpt2-v1, in interpreting and representing human genome sequences. Initially, we meticulously constructed multiple datasets linking genotypes and phenotypes to fine-tune those models for precise DNA sequence classification. Additionally, we evaluate the influence of sequence length on classification results and analyze the impact of feature extraction in the hidden layers of our model using the HERV dataset. To enhance our understanding of phenotype-specific patterns recognized by the model, we perform enrichment, pathogenicity and conservation analyzes of specific motifs in the human endogenous retrovirus (HERV) sequence with high average local representation weight (ALRW) scores. Results: We have constructed multiple genotype-phenotype datasets displaying commendable classification performance in comparison with random genomic sequences, particularly in the HERV dataset, which achieved binary and multi-classification accuracies and F1 values exceeding 0.935 and 0.888, respectively. Notably, the fine-tuning of the HERV dataset not only improved our ability to identify and distinguish diverse information types within DNA sequences but also successfully identified specific motifs associated with neurological disorders and cancers in regions with high ALRW scores. Subsequent analysis of these motifs shed light on the adaptive responses of species to environmental pressures and their co-evolution with pathogens. Conclusions: These findings highlight the potential of pre-trained genomic models in learning DNA sequence representations, particularly when utilizing the HERV dataset, and provide valuable insights for future research endeavors. This study represents an innovative strategy that combines pre-trained genomic model representations with classical methods for analyzing the functionality of genome sequences, thereby promoting cross-fertilization between genomics and artificial intelligence.

Evaluating the representational power of pre-trained DNA language models for regulatory genomics

Genomic Language Models: Opportunities and Challenges

Genomic language model predicts protein co-regulation and function

DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA

L2G: Repurposing Language Models for Genomics Tasks

Advancing regulatory genomics with machine learning

Are Genomic Language Models All You Need? Exploring Genomic Language Models on Protein Downstream Tasks

Exploring the Potentials and Challenges of Using Large Language Models for the Analysis of Transcriptional Regulation of Long Non-coding RNAs

Integrating convolution and self-attention improves language model of human genome for interpreting non-coding regions at base-resolution

Enhancing recognition and interpretation of functional phenotypic sequences through fine-tuning pre-trained genomic models

Deciphering RNA regulation with a foundation language model

Perspective on recent developments and challenges in regulatory and systems genomics

LLM4GRN: Discovering Causal Gene Regulatory Networks with LLMs -- Evaluation through Synthetic Data Generation

A mechanistically interpretable neural network for regulatory genomics

Pharmacotherapy for Disorders of Consciousness: Are ‘Awakening’ Drugs Really a Possibility?

Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types

Pre-training with pseudo-labeling compares favorably with large language models for regulatory sequence prediction

regLM: Designing realistic regulatory DNA with autoregressive language models

How do Large Language Models understand Genes and Cells

Advances in Human Genome Resolution: the Role of Pan-Genomic Strategies and Fine-Tuning Pre-trained Genomic Models

Dissecting and improving gene regulatory network inference using single-cell transcriptome data