Abstract:Quantitative structure-activity relationship (QSAR) methods have been widely applied in drug discovery, lead optimization, toxicity prediction, and regulatory decisions. Despite major advances in algorithms and software, QSAR models have inherent limitations associated with a size and chemical-structure diversity of the training set, experimental error, and many characteristics of structure representation and correlation algorithms. Whereas excellent fit to the training data may be readily attainable, often models fail to predict accurately chemicals that are outside their domain of applicability. A QSAR's utility and, in the case of regulatory decisions, justification for usage increasingly depend on the ability to quantify a model's potential for predicting unknown chemicals with some known degree of certainty. It is never possible to predict an unknown chemical with absolute certainty. Here we report on two QSAR models based on different data sets for classification of chemicals according to their ability to bind to the estrogen receptor. The models were developed by using a novel QSAR method, Decision Forest, which combines the results of multiple heterogeneous but comparable Decision Tree models to produce a consensus prediction. We used an extensive cross-validation process to define an applicability domain for model predictions based on two quantitative measures: prediction confidence and domain extrapolation. Together, these measures quantify the accuracy of each prediction within and outside of the training domain. Despite being based on large and diverse training sets, both QSAR models had poor accuracy for chemicals within the domain of low confidence, whereas good accuracy was obtained for those within the domain of high confidence. For prediction in the high confidence domain, accuracy was inversely proportional to the degree of domain extrapolation. The model with a larger training set of 1,092, compared with 232 for the other, was more accurate in predicting chemicals at larger domain extrapolation, and could be particularly useful for rapidly prioritizing potential endocrine disruptors from large chemical universe.

External validation and prediction employing the predictive squared correlation coefficient test set activity mean vs training set activity mean.

How Precise Are Our Quantitative Structure-Activity Relationship Derived Predictions for New Query Chemicals?

Prediction reliability of QSAR models: an overview of various validation tools

The relationship between model predictive ability and validated method in QSAR/QSPR study

Assessing Qsar Limitations - A Regulatory Perspective

Reliably assessing prediction reliability for high dimensional QSAR data

Why QSAR fails: an empirical evaluation using conventional computational approach.

Assessment of Prediction Confidence and Domain Extrapolation of Two Structure-Activity Relationship Models for Predicting Estrogen Receptor Binding Activity

Development and Evaluation of Conformal Prediction Methods for QSAR

Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

QSAR-Co: An Open Source Software for Developing Robust Multitasking or Multitarget Classification-Based QSAR Models

Local Lazy Regression: Making Use of the Neighborhood to Improve QSAR Predictions

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

An In-Depth Re-Evaluation of Models from the 1st and 2nd Ames/QSAR International Challenge Projects

The coefficient of conformism of a correlative prediction ( CCCP ): Building up reliable nano-QSPRs/QSARs for endpoints of nanoparticles in different experimental conditions encoded via quasi-SMILES

Assessing Model Fit by Cross-Validation

Exploring QSAR models for activity-cliff prediction

Minimum sample size for external validation of a clinical prediction model with a binary outcome

The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation

Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

Do we need different machine learning algorithms for QSAR modeling? A comprehensive assessment of 16 machine learning algorithms on 14 QSAR data sets