Abstract:Pretrained language models have shown promise in analysing nucleotide sequences, yet a versatile model excelling across diverse tasks with a single pretrained weight set remains elusive. Here we introduce RNAErnie, an RNA-focused pretrained model built upon the transformer architecture, employing two simple yet effective strategies. First, RNAErnie enhances pretraining by incorporating RNA motifs as biological priors and introducing motif-level random masking in addition to masked language modelling at base/subsequence levels. It also tokenizes RNA types (for example, miRNA, lnRNA) as stop words, appending them to sequences during pretraining. Second, subject to out-of-distribution tasks with RNA sequences not seen during the pretraining phase, RNAErnie proposes a type-guided fine-tuning strategy that first predicts possible RNA types using an RNA sequence and then appends the predicted type to the tail of sequence to refine feature embedding in a post hoc way. Our extensive evaluation across seven datasets and five tasks demonstrates the superiority of RNAErnie in both supervised and unsupervised learning. It surpasses baselines with up to 1.8% higher accuracy in classification, 2.2% greater accuracy in interaction prediction and 3.3% improved F1 score in structure prediction, showcasing its robustness and adaptability with a unified pretrained foundation. Despite the existence of various pretrained language models for nucleotide sequence analysis, achieving good performance on a broad range of downstream tasks using a single model is challenging. Wang and colleagues develop a pretrained language model specifically optimized for RNA sequence analysis and show that it can outperform state-of-the-art methods in a diverse set of downstream tasks.

Multiple sequence alignment-based RNA language model and its application to structural inference

Accurate RNA 3D Structure Prediction Using a Language Model-Based Deep Learning Approach

Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions

ERNIE-RNA: An RNA Language Model with Structure-enhanced Representations

Language models enable zero-shot prediction of RNA secondary structure including pseudoknots

Deciphering RNA regulation with a foundation language model

Prediction of Multiple Types of RNA Modifications Via Biological Language Model.

RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks

Multi-purpose RNA Language Modelling with Motif-Aware Pretraining and Type-Guided Fine-Tuning

Comprehensive benchmarking of large language models for RNA secondary structure prediction

Prediction of the RNA Tertiary Structure Based on a Random Sampling Strategy and Parallel Mechanism

RNA-TorsionBERT: leveraging language models for RNA 3D torsion angles prediction

Rm-LR: A long-range-based deep learning model for predicting multiple types of RNA modifications

Predicting RNA sequence-structure likelihood via structure-aware deep learning

Predicting Distance matrix with large language models

RNA 3D structure prediction using multiple sequence alignment information

Interpretable Multi-Scale Deep Learning for RNA Methylation Analysis across Multiple Species

Diverse Database and Machine Learning Model to Narrow the Generalization Gap in RNA Structure Prediction

Machine learning in RNA structure prediction: Advances and challenges

ProtRNA: A Protein-derived RNA Language Model by Cross-Modality Transfer Learning