Benchmarking structure-based three-dimensional molecular generative models using GenBench3D: ligand conformation quality matters

Benoit Baillif,Jason Cole,Patrick McCabe,Andreas Bender

2024-07-05

Abstract:Three-dimensional (3D) deep molecular generative models offer the advantage of goal-directed generation based on 3D-dependent properties, such as binding affinity for structure-based design within binding pockets. Traditional benchmarks created to evaluate SMILES or molecular graphs generators, such as GuacaMol or MOSES, are limited to evaluate 3D generators as they do not assess the quality of the generated molecular conformation. In this work, we hence developed GenBench3D, which implements a new benchmark for models producing molecules within a binding pocket. Our main contribution is the Validity3D metric, evaluating the conformation quality using the likelihood of bond lengths and valence angles based on reference values observed in the Cambridge Structural Database. The LiGAN, 3D-SBDD, Pocket2Mol, TargetDiff, DiffSBDD and ResGen models were benchmarked. We show that only between 0% and 11% of generated molecules have valid conformations. Performing local relaxation of generated molecules in the pocket considerably improved the Validity3D for all models by a minimum increase of 40%. For LiGAN, 3D-SBDD, or TargetDiff, the set of valid relaxed molecules shows on average higher Vina score (i.e. worse) than the set of raw generated molecules, indicating that the binding affinity of raw generated molecules might be overestimated. Using the other scoring functions, that give higher importance to ligand strain, only yield improved scores when using valid relaxed molecules. Using valid relaxed molecules, TargetDiff and Pocket2Mol show better median Vina, Glide and Gold PLP scores than other models. We have publicly released GenBench3D on GitHub for broader use: <a class="link-external link-https" href="https://github.com/bbaillif/genbench3d" rel="external noopener nofollow">this https URL</a>

Quantitative Methods,Biomolecules

What problem does this paper attempt to address?

The paper primarily aims to address the evaluation problem of 3D molecular generative models based on structural foundations, particularly focusing on the quality of generated molecular conformations. Specifically, the research team developed a new benchmarking framework—GenBench3D, for evaluating 3D generative models.

The core issues addressed in the paper include:
1. **Limitations of existing benchmarks**: Traditional benchmarks like GuacaMol or MOSES are only suitable for evaluating SMILES strings or molecular graph generators and do not consider the specific 3D conformation quality of generated molecules.
2. **Validity of molecular conformations**: To overcome the above limitations, the authors introduced a new evaluation metric—Validity3D, aimed at quantifying the geometric validity of generated molecular conformations, i.e., the reasonableness of bond lengths and bond angles in comparison to known reference values.
3. **Evaluation of generated molecules**: The paper not only evaluates the conformation quality of generated molecules but also utilizes different scoring functions (such as Vina, Glide, and Gold PLP) to estimate the binding affinity of generated molecules compared to the original ligands.
4. **Impact of local optimization**: Furthermore, the researchers conducted local optimization (relaxation), which involves locally minimizing the generated molecules to improve their conformations, and observed how this process affects the validity of molecular conformations and the estimated binding affinity.

Through these efforts, the paper aims to establish a comprehensive evaluation system that can more accurately assess the capabilities of 3D molecular generative models, particularly their potential in applications such as drug design.