Abstract:Machine learning has gained popularity for predicting molecular properties based on molecular structure. This study explores the uncertainty estimates of neural fingerprint-based models by comparing pure graph neural networks (GNN) to classical machine learning algorithms combined with neural fingerprints. We investigate the advantage of extracting the neural fingerprint from the GNN and integrating it into a method known for producing better-calibrated probability estimates. Comparisons are made using three classical machine learning methods and the Chemprop model, considering different molecular representations and calibration techniques. We utilize 19 datasets from Toxcast, reflecting real-world scenarios with balanced accuracies ranging from 0.6 to 0.8. Results demonstrate that neural fingerprints combined with classical machine learning methods exhibit a slight decrease in prediction performance compared to the native Chemprop model. However, these models provide significantly improved uncertainty estimates. Notably, uncertainty estimates of neural fingerprint-based methods remain relatively robust for molecules dissimilar to the training set. This suggests that methods like random forest with neural fingerprints can deliver strong prediction performance and reliable uncertainty estimates. When considering both performance and uncertainty, the calibrated Chemprop model and the combination of neural fingerprints with random forest or support vector classifier (SVC) yield comparable results. Surprisingly, the SVC method shows promising performance when combined with neural or count fingerprints. These findings are particularly relevant in real-world industrial projects where accurate predictions and reliable uncertainty estimates are crucial.

Chemoinformatic regression methods and their applicability domain

A Statistical View of Some Chemometrics Regression Tools

Characterizing Uncertainty in Machine Learning for Chemistry

Regression Models for Machine Learning

Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry

How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?

Methods for comparing uncertainty quantifications for material property predictions

Error Assessment of Computational Models in Chemistry

Determining Domain of Machine Learning Models using Kernel Density Estimates: Applications in Materials Property Prediction

A Combinational Strategy of Model Disturbance and Outlier Comparison to Define Applicability Domain in Quantitative Structural Activity Relationship

Rethinking the applicability domain analysis in QSAR models

Uncertainty Quantification Metrics for Deep Regression

Robust multivariate methods in Chemometrics

A hybrid framework for improving uncertainty quantification in deep learning-based QSAR regression modeling

Analysis of uncertainty of neural fingerprint-based models

Evolution of Support Vector Machine and Regression Modeling in Chemoinformatics and Drug Discovery

Robustness Evaluation of Regression Tasks with Skewed Domain Preferences

chemmodlab: A Cheminformatics Modeling Laboratory for Fitting and Assessing Machine Learning Models

Quantitative Regression Models For The Prediction Of Chemical Properties By An Efficient Workflow

Large-scale evaluation of k-fold cross-validation ensembles for uncertainty estimation

A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning