Abstract:Advances in deep neural network (DNN)-based molecular property prediction have recently led to the development of models of remarkable accuracy and generalization ability, with graph convolutional neural networks (GCNNs) reporting state-of-the-art performance for this task. However, some challenges remain, and one of the most important that needs to be fully addressed concerns uncertainty quantification. DNN performance is affected by the volume and the quality of the training samples. Therefore, establishing when and to what extent a prediction can be considered reliable is just as important as outputting accurate predictions, especially when out-of-domain molecules are targeted. Recently, several methods to account for uncertainty in DNNs have been proposed, most of which are based on approximate Bayesian inference. Among these, only a few scale to the large data sets required in applications. Evaluating and comparing these methods has recently attracted great interest, but results are generally fragmented and absent for molecular property prediction. In this paper, we quantitatively compare scalable techniques for uncertainty estimation in GCNNs. We introduce a set of quantitative criteria to capture different uncertainty aspects and then use these criteria to compare MC-dropout, Deep Ensembles, and bootstrapping, both theoretically in a unified framework that separates aleatoric/epistemic uncertainty and experimentally on public data sets. Our experiments quantify the performance of the different uncertainty estimation methods and their impact on uncertainty-related error reduction. Our findings indicate that Deep Ensembles and bootstrapping consistently outperform MC-dropout, with different context-specific pros and cons. Our analysis leads to a better understanding of the role of aleatoric/epistemic uncertainty, also in relation to the target data set features, and highlights the challenge posed by out-of-domain uncertainty.

Methods for comparing uncertainty quantifications for material property predictions

Materials Property Prediction with Uncertainty Quantification: A Benchmark Study

Approaches for Uncertainty Quantification of AI-predicted Material Properties: A Comparison

Uncertainty Quantification Using Neural Networks for Molecular Property Prediction

Characterizing Uncertainty in Machine Learning for Chemistry

Uncertainty Prediction for Machine Learning Models of Material Properties

Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantification

Explainable uncertainty quantifications for deep learning-based molecular property prediction

Evaluating Scalable Uncertainty Estimation Methods for Deep Learning-Based Molecular Property Prediction

Uncertainty Quantification in Molecular Property Prediction Through Spherical Mixture Density Networks.

Fast and Accurate Uncertainty Estimation in Chemical Machine Learning

Evaluating uncertainty-based active learning for accelerating the generalization of molecular property prediction

Fast Uncertainty Estimates in Deep Learning Interatomic Potentials

A hybrid framework for improving uncertainty quantification in deep learning-based QSAR regression modeling

Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry

Uncertainty Quantification and Propagation in Atomistic Machine Learning

Improved Uncertainty Estimation of Graph Neural Network Potentials Using Engineered Latent Space Distances

Statistical methods for resolving poor uncertainty quantification in machine learning interatomic potentials

Bayesian strategies for uncertainty quantification of the thermodynamic properties of materials

Uncertainty quantification by direct propagation of shallow ensembles

A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning