Abstract:While a multitude of deep generative models have recently emerged there exists no best practice for their practically relevant validation. On the one hand, novel de novo-generated molecules cannot be refuted by retrospective validation (so that this type of validation is biased); but on the other hand prospective validation is expensive and then often biased by the human selection process. In this case study, we frame retrospective validation as the ability to mimic human drug design, by answering the following question: Can a generative model trained on early-stage project compounds generate middle/late-stage compounds de novo? To this end, we used experimental data that contains the elapsed time of a synthetic expansion following hit identification from five public (where the time series was pre-processed to better reflect realistic synthetic expansions) and six in-house project datasets, and used REINVENT as a widely adopted RNN-based generative model. After splitting the dataset and training REINVENT on early-stage compounds, we found that rediscovery of middle/late-stage compounds was much higher in public projects (at 1.60%, 0.64%, and 0.21% of the top 100, 500, and 5,000 scored generated compounds) than in in-house projects (where the values were 0.00%, 0.03%, and 0.04%, respectively). Similarly, average single nearest neighbour similarity between early- and middle/late-stage compounds in public projects was higher between active compounds than inactive compounds; however, for in-house projects the converse was true, which makes rediscovery (if so desired) more difficult. We hence show that the generative model recovers very few middle/late-stage compounds from real-world drug discovery projects, highlighting the fundamental difference between purely algorithmic design and drug discovery as a real-world process. Evaluating de novo compound design approaches appears, based on the current study, difficult or even impossible to do retrospectively. "Scientific Contribution" This contribution hence illustrates aspects of evaluating the performance of generative models in a real-world setting which have not been extensively described previously and which hopefully contribute to their further future development.

Quantitative evaluation of molecular generation performance of graph-based GANs

Graph-Based Generative Adversarial Networks for Molecular Generation with Noise Diffusion.

Advances in Deep Learning-Based 3D Molecular Generative Models

Enhancing molecular design efficiency: Uniting language models and generative networks with genetic algorithms

Quantum Generative Models for Small Molecule Drug Discovery

STAGAN: an Approach for Improve the Stability of Molecular Graph Generation Based on Generative Adversarial Networks.

Exploring the Advantages of Quantum Generative Adversarial Networks in Generative Chemistry

On The Difficulty of Validating Molecular Generative Models Realistically: A Case Study on Public and Proprietary Data

Molecular Generative Adversarial Network with Multi-Property Optimization

Will More Expressive Graph Neural Networks do Better on Generative Tasks?

QGAN: Quantized Generative Adversarial Networks.

Quantization of Generative Adversarial Networks for Efficient Inference: a Methodological Study

On the Evaluation of Generative Adversarial Networks By Discriminative Models

Understanding GANs: fundamentals, variants, training challenges, applications, and open problems

Research on Data Generation Model Based on Improved SeqGAN

Pros and cons of GAN evaluation measures

When Molecular GAN Meets Byte-Pair Encoding

Survey of Quantum Generative Adversarial Networks (QGAN) to Generate Images

MedGAN: optimized generative adversarial network with graph convolutional networks for novel molecule design

Revisiting the Evaluation of Image Synthesis with GANs

Game of GANs: Game-Theoretical Models for Generative Adversarial Networks