Relationship between prediction accuracy and uncertainty in compound potency prediction using deep neural networks and control models

Jannik P. Roth,Jürgen Bajorath

DOI: https://doi.org/10.1038/s41598-024-57135-6

IF: 4.6

2024-03-20

Scientific Reports

Abstract:The assessment of prediction variance or uncertainty contributes to the evaluation of machine learning models. In molecular machine learning, uncertainty quantification is an evolving area of research where currently no standard approaches or general guidelines are available. We have carried out a detailed analysis of deep neural network variants and simple control models for compound potency prediction to study relationships between prediction accuracy and uncertainty. For comparably accurate predictions obtained with models of different complexity, highly variable prediction uncertainties were detected using different metrics. Furthermore, a strong dependence of prediction characteristics and uncertainties on potency levels of test compounds was observed, often leading to over- or under-confident model decisions with respect to the expected variance of predictions. Moreover, neural network models responded very differently to training set modifications. Taken together, our findings indicate that there is only little, if any correlation between compound potency prediction accuracy and uncertainty, especially for deep neural network models, when predictions are assessed on the basis of currently used metrics for uncertainty quantification.

multidisciplinary sciences

What problem does this paper attempt to address?

The paper investigates the relationship between prediction accuracy and uncertainty in compound potency prediction using deep neural networks and control models. The primary problem addressed is the lack of standard approaches or general guidelines for uncertainty quantification (UQ) in molecular machine learning, particularly regarding compound potency prediction. The authors conduct a detailed analysis of deep neural network variants and simple control models to study the relationship between prediction accuracy and uncertainty. Key findings include: 1. **Highly Variable Prediction Uncertainties**: For predictions of similar accuracy obtained with models of different complexities, the authors observe highly variable prediction uncertainties when using different metrics. 2. **Potency Level Dependence**: There is a strong dependence of prediction characteristics and uncertainties on the potency levels of test compounds. This often leads to over- or under-confident model decisions with respect to the expected variance of predictions. 3. **Response to Training Set Modifications**: Neural network models respond very differently to modifications in the training set. For example, changes in the distribution of training data can significantly affect the model's performance and uncertainty estimates. 4. **Correlation Between Accuracy and Uncertainty**: The findings suggest that there is little to no correlation between compound potency prediction accuracy and uncertainty, especially in the context of different model complexities and training set distributions.

Relationship between prediction accuracy and uncertainty in compound potency prediction using deep neural networks and control models

Evaluation of multi-target deep neural network models for compound potency prediction under increasingly challenging test conditions

Analysis of uncertainty of neural fingerprint-based models

Uncertainty Quantification Using Neural Networks for Molecular Property Prediction

Understanding the Limitations of Deep Models for Molecular Property Prediction: Insights and Solutions.

Evaluating Scalable Uncertainty Estimation Methods for Deep Learning-Based Molecular Property Prediction

Characterizing Uncertainty in Machine Learning for Chemistry

A deep learning based multi-model approach for predicting drug-like chemical compound's toxicity

Predictive Uncertainty Quantification with Compound Density Networks

Uncertainty Qualification for Deep Learning-Based Elementary Reaction Property Prediction

Scalable Bayesian Uncertainty Quantification for Neural Network Potentials: Promise and Pitfalls

Prediction of physicochemical properties based on neural network modelling

A Bayesian graph convolutional network for reliable prediction of molecular properties with uncertainty quantification

Why Deep Models Often cannot Beat Non-deep Counterparts on Molecular Property Prediction?

An Analysis of Proteochemometric and Conformal Prediction Machine Learning Protein-Ligand Binding Affinity Models

Accurate Clinical Toxicity Prediction using Multi-task Deep Neural Nets and Contrastive Molecular Explanations

A Deep Neural Network -- Mechanistic Hybrid Model to Predict Pharmacokinetics in Rat

Complex machine learning model needs complex testing: Examining predictability of molecular binding affinity by a graph neural network

Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Models

A deep neural network: mechanistic hybrid model to predict pharmacokinetics in rat

On the Uncertainty Estimates of Equivariant-Neural-Network-Ensembles Interatomic Potentials