Abstract:BackgroundRNA molecules play diverse functional and structural roles in cells. They function as messengers for transferring genetic information from DNA to proteins, as the primary genetic material in many viruses, as catalysts (ribozymes) important for protein synthesis and RNA processing, and as essential and ubiquitous regulators of gene expression in living organisms. Many of these functions depend on precisely orchestrated interactions between RNA molecules and specific proteins in cells. Understanding the molecular mechanisms by which proteins recognize and bind RNA is essential for comprehending the functional implications of these interactions, but the recognition ‘code’ that mediates interactions between proteins and RNA is not yet understood. Success in deciphering this code would dramatically impact the development of new therapeutic strategies for intervening in devastating diseases such as AIDS and cancer. Because of the high cost of experimental determination of protein-RNA interfaces, there is an increasing reliance on statistical machine learning methods for training predictors of RNA-binding residues in proteins. However, because of differences in the choice of datasets, performance measures, and data representations used, it has been difficult to obtain an accurate assessment of the current state of the art in protein-RNA interface prediction.ResultsWe provide a review of published approaches for predicting RNA-binding residues in proteins and a systematic comparison and critical assessment of protein-RNA interface residue predictors trained using these approaches on three carefully curated non-redundant datasets. We directly compare two widely used machine learning algorithms (Naïve Bayes (NB) and Support Vector Machine (SVM)) using three different data representations in which features are encoded using either sequence- or structure-based windows. Our results show that (i) Sequence-based classifiers that use a position-specific scoring matrix (PSSM)-based representation (PSSMSeq) outperform those that use an amino acid identity based representation (IDSeq) or a smoothed PSSM (SmoPSSMSeq); (ii) Structure-based classifiers that use smoothed PSSM representation (SmoPSSMStr) outperform those that use PSSM (PSSMStr) as well as sequence identity based representation (IDStr). PSSMSeq classifiers, when tested on an independent test set of 44 proteins, achieve performance that is comparable to that of three state-of-the-art structure-based predictors (including those that exploit geometric features) in terms of Matthews Correlation Coefficient (MCC), although the structure-based methods achieve substantially higher Specificity (albeit at the expense of Sensitivity) compared to sequence-based methods. We also find that the expected performance of the classifiers on a residue level can be markedly different from that on a protein level. Our experiments show that the classifiers trained on three different non-redundant protein-RNA interface datasets achieve comparable cross-validation performance. However, we find that the results are significantly affected by differences in the distance threshold used to define interface residues.ConclusionsOur results demonstrate that protein-RNA interface residue predictors that use a PSSM-based encoding of sequence windows outperform classifiers that use other encodings of sequence windows. While structure-based methods that exploit geometric features can yield significant increases in the Specificity of protein-RNA interface residue predictions, such increases are offset by decreases in Sensitivity. These results underscore the importance of comparing alternative methods using rigorous statistical procedures, multiple performance measures, and datasets that are constructed based on several alternative definitions of interface residues and redundancy cutoffs as well as including evaluations on independent test sets into the comparisons.

Assessing the accuracy of prediction algorithms for classification: an overview

Prediction of Functional Class of Proteins and Peptides Irrespective of Sequence Homology by Support Vector Machines.

Making Early Predictions of the Accuracy of Machine Learning Applications

Prediction of Interaction Between Enzymes and Small Molecules in Metabolic Pathways Through Integrating Multiple Classifiers.

Protein Classification using Machine Learning and Statistical Techniques: A Comparative Analysis

Prediction of membrane protein types in a hybrid space.

Assessing the Performances of Protein Function Prediction Algorithms from the Perspectives of Identification Accuracy and False Discovery Rate

Prediction of protein structural classes for low-homology sequences based on predicted secondary structure

A method of classification accuracy calculation for cost sensitive algorithms

An empirical study of classification algorithm evaluation for financial risk prediction

Understanding Prediction Discrepancies in Machine Learning Classifiers

Mistakes in validating the accuracy of a prediction classifier in high-dimensional but small-sample microarray data

Does the evaluation stand up to evaluation? A first-principle approach to the evaluation of classifiers

Protein-RNA interface residue prediction using machine learning: an assessment of the state of the art

Classifier Calibration: A survey on how to assess and improve predicted class probabilities

Comparative Analysis of Predictive Algorithms for Performance Measurement

How Good is Prediction of Protein Structural Class by the Component-Coupled Method?

An In-Depth Examination of Risk Assessment in Multi-Class Classification Algorithms

Is it better to combine predictions?

Efficient and Interpretable Prediction of Protein Functional Classes by Correspondence Analysis and Compact Set Relations.

Classifier comparison using precision