Are we fitting data or noise? Analysing the predictive power of commonly used datasets in drug-, materials-, and molecular-discovery.

Philip Biggin,Flaviu Cipcigan,Daniel Crusius

DOI: https://doi.org/10.26434/chemrxiv-2024-z0pz7

2024-05-08

Abstract:Data-driven techniques for establishing quantitative structure property relations are a pillar of modern materials and molecular discovery. Fuelled by the recent progress in deep learning methodology and the abundance of new algorithms, it is tempting to chase benchmarks and incrementally build ever more capable machine learning (ML) models. While model evaluation has made significant progress, the intrinsic limitations arising from the underlying experimental data are often overlooked. In the chemical sciences data collection is costly, thus datasets are small and experimental errors can be significant. These limitations of such datasets affect their predictive power, a fact that is rarely considered in a quantitative way. In this study, we analyse commonly used ML datasets for regression and classification from drug discovery, molecular discovery, and materials discovery. We derived maximum and realistic performance bounds for nine such datasets by introducing noise based on estimated or actual experimental errors. We then compared the estimated performance bounds to the reported performance of leading ML models in the literature. Out of the nine datasets and corresponding ML models considered, four were identified to have reached or surpassed dataset performance limitations and thus, they may potentially be fitting noise. More generally, we systematically examine how data range, the magnitude of experimental error, and the number of data points influence dataset performance bounds. Alongside this paper, we release the Python package NoiseEstimator and provide a web- based application for computing realistic performance bounds. This study and the resulting tools will help practitioners in the field understand the limitations of datasets and set realistic expectations for ML model performance. This work stands as a reference point, offering analysis and tools to guide development of future ML models in the chemical sciences.

Chemistry

What problem does this paper attempt to address?

This paper analyzes the limitations of commonly used datasets in chemistry and materials science in terms of the predictive capability of machine learning (ML) models. The study determines the maximum and realistic performance boundaries of the datasets by introducing noise based on experimental errors, and compares the reported performance of leading ML models. The study finds that four datasets may have reached or exceeded the performance limitations caused by data noise, implying that these models may be overfitting the noise. The paper emphasizes the impact of considering data noise on model performance and provides the Python package NoiseEstimator and an online application to compute performance boundaries, helping researchers understand the limitations of datasets and set reasonable expectations for model performance.

Are we fitting data or noise? Analysing the predictive power of commonly used datasets in drug-, materials-, and molecular-discovery.

Are we fitting data or noise? Analysing the predictive power of commonly used datasets in drug-, materials-, and molecular-discovery.

Understanding the Limitations of Deep Models for Molecular Property Prediction: Insights and Solutions.

Current Methods for Drug Property Prediction in the Real World

Denoising Drug Discovery Data for Improved ADMET Property Prediction

Audacity of huge: overcoming challenges of data scarcity and data quality for machine learning in computational materials discovery

Learning data efficient coarse-grained molecular dynamics from forces and noise

Quantifying the performance of machine learning models in materials discovery

Machine Learning Small Molecule Properties in Drug Discovery

Data-Driven Materials Discovery and Synthesis using Machine Learning Methods

A call for an industry-led initiative to critically assess machine learning for real-world drug discovery

A strategy to apply machine learning to small datasets in materials science

From Prediction to Action: Critical Role of Performance Estimation for Machine-Learning-Driven Materials Discovery

A decision theoretic approach to model evaluation in computational drug discovery

Performance Insights for Small Molecule Drug Discovery Models: Data Scaling, Multitasking, and Generalization

A systematic study of key elements underlying molecular property prediction

A critical examination of robustness and generalizability of machine learning prediction of materials properties

Deep-learning: investigating deep neural networks hyper-parameters and comparison of performance to shallow methods for modeling bioactivity data

Unraveling Key Elements Underlying Molecular Property Prediction: A Systematic Study

Distance-based Analysis of Machine Learning Prediction Reliability for Datasets in Materials Science and Other Fields

Extrapolative prediction of small-data molecular property using quantum mechanics-assisted machine learning