Abstract:While the use of deep learning in drug discovery is gaining increasing attention, the lack of methods to compute reliable errors in prediction for Neural Networks prevents their application to guide decision making in domains where identifying unreliable predictions is essential, e.g., precision medicine. Here, we present a framework to compute reliable errors in prediction for Neural Networks using Test-Time Dropout and Conformal Prediction. Specifically, the algorithm consists of training a single Neural Network using dropout, and then applying it N times to both the validation and test sets, also employing dropout in this step. Therefore, for each instance in the validation and test sets an ensemble of predictions are generated. The residuals and absolute errors in prediction for the validation set are then used to compute prediction errors for the test set instances using Conformal Prediction. We show using 24 bioactivity data sets from ChEMBL 23 that Dropout Conformal Predictors are valid (i.e., the fraction of instances whose true value lies within the predicted interval strongly correlates with the confidence level) and efficient, as the predicted confidence intervals span a narrower set of values than those computed with Conformal Predictors generated using Random Forest (RF) models. Lastly, we show in retrospective virtual screening experiments that dropout and RF-based Conformal Predictors lead to comparable retrieval rates of active compounds. Overall, we propose a computationally efficient framework (as only N extra forward passes are required in addition to training a single network) to harness Test-Time Dropout and the Conformal Prediction framework, which is generally applicable to generate reliable prediction errors for Deep Neural Networks in drug discovery and beyond.

Testing Spintronics Implemented Monte Carlo Dropout-Based Bayesian Neural Networks

Spatial-SpinDrop: Spatial Dropout-based Binary Bayesian Neural Network with Spintronics Implementation

Scalable Spintronics-based Bayesian Neural Network for Uncertainty Estimation

Uncertainty propagation for dropout-based Bayesian neural networks

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Hardware-Aware Neural Dropout Search for Reliable Uncertainty Prediction on FPGA

DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks

Efficient Uncertainty Estimation in Spiking Neural Networks via MC-dropout

Enhancing Dropout-based Bayesian Neural Networks with Multi-Exit on FPGA

Dropout injection at test time for post hoc uncertainty quantification in neural networks

Single-shot Bayesian approximation for neural networks

Few-Shot Testing: Estimating Uncertainty of Memristive Deep Neural Networks Using One Bayesian Test Vector

Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout

Dropout neuronal unit with tunable probability based on NbO

Concurrent Self-testing of Neural Networks Using Uncertainty Fingerprint

Applying Monte Carlo Dropout to Quantify the Uncertainty of Skip Connection-Based Convolutional Neural Networks Optimized by Big Data

Two-dimensional materials-based probabilistic synapses and reconfigurable neurons for measuring inference uncertainty using Bayesian neural networks

Scalable and Efficient Methods for Uncertainty Estimation and Reduction in Deep Learning

Estimation of Small Failure Probability Based on Adaptive Subset Simulation and Deep Neural Network

Bayesian neural networks for uncertainty quantification in remaining useful life prediction of systems with sensor monitoring