From Theory to Therapy: Reframing SBDD Model Evaluation via Practical Metrics

Bowen Gao,Haichuan Tan,Yanwen Huang,Minsi Ren,Xiao Huang,Wei-Ying Ma,Ya-Qin Zhang,Yanyan Lan

2024-06-13

Abstract:Recent advancements in structure-based drug design (SBDD) have significantly enhanced the efficiency and precision of drug discovery by generating molecules tailored to bind specific protein pockets. Despite these technological strides, their practical application in real-world drug development remains challenging due to the complexities of synthesizing and testing these molecules. The reliability of the Vina docking score, the current standard for assessing binding abilities, is increasingly questioned due to its susceptibility to overfitting. To address these limitations, we propose a comprehensive evaluation framework that includes assessing the similarity of generated molecules to known active compounds, introducing a virtual screening-based metric for practical deployment capabilities, and re-evaluating binding affinity more rigorously. Our experiments reveal that while current SBDD models achieve high Vina scores, they fall short in practical usability metrics, highlighting a significant gap between theoretical predictions and real-world applicability. Our proposed metrics and dataset aim to bridge this gap, enhancing the practical applicability of future SBDD models and aligning them more closely with the needs of pharmaceutical research and development.

Biomolecules,Machine Learning

What problem does this paper attempt to address?

This paper focuses on the evaluation problem of Structure-Based Drug Design (SBDD) models. Despite recent advancements in SBDD, which have improved the efficiency and accuracy of drug discovery, applying these models to practical drug development still faces challenges such as the complexity of synthesizing and testing new molecules. Currently, the Vina docking score serves as the standard for evaluating molecular binding capabilities, but its reliability is questioned as it may overestimate performance due to overfitting.

To address these issues, the paper proposes a comprehensive evaluation framework that includes three aspects:
1. Comparing the similarity between molecules and known active compounds and FDA-approved drugs to assess their feasibility as potential drug candidates.
2. Introducing indicators based on virtual screening to measure the practical deployability of molecules.
3. Evaluating binding affinity more rigorously to avoid excessive reliance on Vina scores.

Experimental results demonstrate that although current SBDD models perform well in terms of Vina scores, there is a significant gap in terms of practical usability indicators. The paper emphasizes the gap between theoretical predictions and real-world applications, and proposes new evaluation standards and datasets aimed at enhancing the practicality and alignment with pharmaceutical research and development needs for future SBDD models.