Holistic chemical evaluation reveals pitfalls in reaction prediction models

Victor Sabanza Gil,Andres M. Bran,Malte Franke,Remi Schlama,Jeremy S. Luterbacher,Philippe Schwaller
2023-12-14
Abstract:The prediction of chemical reactions has gained significant interest within the machine learning community in recent years, owing to its complexity and crucial applications in chemistry. However, model evaluation for this task has been mostly limited to simple metrics like top-k accuracy, which obfuscates fine details of a model's limitations. Inspired by progress in other fields, we propose a new assessment scheme that builds on top of current approaches, steering towards a more holistic evaluation. We introduce the following key components for this goal: CHORISO, a curated dataset along with multiple tailored splits to recreate chemically relevant scenarios, and a collection of metrics that provide a holistic view of a model's advantages and limitations. Application of this method to state-of-the-art models reveals important differences on sensitive fronts, especially stereoselectivity and chemical out-of-distribution generalization. Our work paves the way towards robust prediction models that can ultimately accelerate chemical discovery.
Chemical Physics,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the deficiencies in current evaluation methods for chemical reaction prediction models. Specifically, existing model evaluations mainly rely on simple metrics such as top - k accuracy, which masks the limitations of models in complex chemical reaction prediction tasks. The paper proposes a new comprehensive evaluation scheme, aiming to more comprehensively assess the strengths and limitations of these models, especially in aspects such as stereoselectivity and out - of - chemical - distribution generalization ability. By introducing the carefully curated dataset CHORISO and a series of customized evaluation metrics, this research reveals significant differences among existing models in sensitive areas and paves the way for the development of more powerful prediction models, ultimately accelerating the process of chemical discovery.