Abstract:Background: Decoding human genomic sequences requires comprehensive analysis of DNA sequence functionality. Through computational and experimental approaches, researchers have studied the genotype-phenotype relationship and generate important datasets that help unravel complicated genetic blueprints. Thus, the recently developed artificial intelligence methods can be used to interpret the functions of those DNA sequences. Methods: This study explores the use of deep learning, particularly pre-trained genomic models like DNA_bert_6 and human_gpt2-v1, in interpreting and representing human genome sequences. Initially, we meticulously constructed multiple datasets linking genotypes and phenotypes to fine-tune those models for precise DNA sequence classification. Additionally, we evaluate the influence of sequence length on classification results and analyze the impact of feature extraction in the hidden layers of our model using the HERV dataset. To enhance our understanding of phenotype-specific patterns recognized by the model, we perform enrichment, pathogenicity and conservation analyzes of specific motifs in the human endogenous retrovirus (HERV) sequence with high average local representation weight (ALRW) scores. Results: We have constructed multiple genotype-phenotype datasets displaying commendable classification performance in comparison with random genomic sequences, particularly in the HERV dataset, which achieved binary and multi-classification accuracies and F1 values exceeding 0.935 and 0.888, respectively. Notably, the fine-tuning of the HERV dataset not only improved our ability to identify and distinguish diverse information types within DNA sequences but also successfully identified specific motifs associated with neurological disorders and cancers in regions with high ALRW scores. Subsequent analysis of these motifs shed light on the adaptive responses of species to environmental pressures and their co-evolution with pathogens. Conclusions: These findings highlight the potential of pre-trained genomic models in learning DNA sequence representations, particularly when utilizing the HERV dataset, and provide valuable insights for future research endeavors. This study represents an innovative strategy that combines pre-trained genomic model representations with classical methods for analyzing the functionality of genome sequences, thereby promoting cross-fertilization between genomics and artificial intelligence.

Improving language model of human genome for DNA–protein binding prediction based on task-specific pre-training

Toward Understanding BERT-Like Pre-Training for DNA Foundation Models

Predictive Recognition of DNA-binding Proteins Based on Pre-trained Language Model BERT

BERT-TFBS: a novel BERT-based model for predicting transcription factor binding sites by transfer learning

Rethinking the BERT-like Pretraining for DNA Sequences.

Advancing Transcription Factor Binding Site Prediction Using DNA Breathing Dynamics and Sequence Transformers via Cross Attention

Interpretable improving prediction performance of general protein language model by domain-adaptive pretraining on DNA-binding protein

DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome

DNA breathing integration with deep learning foundational model advances genome-wide binding prediction of human transcription factors

Biologically relevant transfer learning improves transcription factor binding prediction

BertSNR: an interpretable deep learning framework for single-nucleotide resolution identification of transcription factor binding sites based on DNA language model

Transfer learning and DNA language models enhance transcription factor binding predictions

Advancing Protein-DNA Binding Site Prediction: Integrating Sequence Models and Machine Learning Classifiers

Deciphering the Language of Protein-DNA Interactions: A Deep Learning Approach Combining Contextual Embeddings and Multi-Scale Sequence Modeling

Predicting the sequence specificities of DNA-binding proteins by DNA Fine-tuned Language Model with decaying learning rates

ChromBERT: Uncovering Chromatin State Motifs in the Human Genome Using a BERT-based Approach

Protein–DNA binding sites prediction based on pre-trained protein language model and contrastive learning

Protein-DNA binding sites prediction based on pre-trained protein language model and contrastive learning

Benchmarking DNA Foundation Models for Genomic Sequence Classification

Enhancing recognition and interpretation of functional phenotypic sequences through fine-tuning pre-trained genomic models