Abstract:With large amounts of unlabeled RNA sequences data produced by high-throughput sequencing technologies, pre-trained RNA language models have been developed to estimate semantic space of RNA molecules, which facilities the understanding of grammar of RNA language. However, existing RNA language models overlook the impact of structure when modeling the RNA semantic space, resulting in incomplete feature extraction and suboptimal performance across various downstream tasks. In this study, we developed a RNA pre-trained language model named ERNIE-RNA ( nhanced eprese tations with base-pa ring r striction for modeling) based on a modified BERT (Bidirectional Encoder Representations from Transformers) by incorporating base-pairing restriction with no MSA (Multiple Sequence Alignment) information. We found that the attention maps from ERNIE-RNA with no fine-tuning are able to capture RNA structure in the zero-shot experiment more precisely than conventional methods such as fine-tuned RNAfold and RNAstructure, suggesting that the ERNIE-RNA can provide comprehensive RNA structural representations. Furthermore, ERNIE-RNA achieved SOTA (state-of-the-art) performance after fine-tuning for various downstream tasks, including RNA structural and functional predictions. In summary, our ERNIE-RNA model provides general features which can be widely and effectively applied in various subsequent research tasks. Our results indicate that introducing key knowledge-based prior information in the BERT framework may be a useful strategy to enhance the performance of other language models.

RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks

RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks

Diverse Database and Machine Learning Model to Narrow the Generalization Gap in RNA Structure Prediction

Language models enable zero-shot prediction of RNA secondary structure including pseudoknots

Comprehensive benchmarking of large language models for RNA secondary structure prediction

Multiple sequence alignment-based RNA language model and its application to structural inference

Accurate RNA 3D Structure Prediction Using a Language Model-Based Deep Learning Approach

Deciphering RNA regulation with a foundation language model

RNA language models predict mutations that improve RNA function

GenerRNA: A generative pre-trained language model for RNA design

ProtRNA: A Protein-derived RNA Language Model by Cross-Modality Transfer Learning

ERNIE-RNA: An RNA Language Model with Structure-enhanced Representations

GenerRNA: A generative pre-trained language model for de novo RNA design

DGRNA: a long-context RNA foundation model with bidirectional attention Mamba2

Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions

UNI-RNA: UNIVERSAL PRE-TRAINED MODELS REVOLUTIONIZE RNA RESEARCH

Rm-LR: A long-range-based deep learning model for predicting multiple types of RNA modifications

Single-sequence protein-RNA complex structure prediction by geometric attention-enabled pairing of biological language models

Ribonanza: deep learning of RNA structure through dual crowdsourcing

Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Models

RNA-GPT: Multimodal Generative System for RNA Sequence Understanding