Abstract:Machine learning (ML) is driving exciting innovations in drug discovery, but we need to be mindful of the circumstances that set this application apart. Unlike other fields with fit-for-purpose datasets consisting of millions of examples, published datasets in drug discovery are classically heterogeneous, imbalanced, noisy and expensive to generate 1 . Furthermore, the applications of ML in drug discovery are numerous, require familiarity with several scientific disciplines, and inform high-stakes decisions, such as expensive or time-consuming experiments. The absence of standardized, domain-appropriate datasets, guidelines and tools for the evaluation and comparison of methods has led to a growing gap between perceived progress and real-world impact, which is delaying the adoption of ML in drug discovery. To bridge this gap, we believe that the unique expertise of scientists in the industry, who operate in real-world contexts, will be essential in developing benchmarking protocols tailored to drug discovery. To that end, we already formed a unique collaboration between representatives from ten biotech and pharmaceutical companies, but we believe that an open-science, cross-industry and interdisciplinary effort is needed to tackle such grand challenges. Fit-for-purpose benchmarks are powerful instruments to direct the ML community towards more impactful research and to unlock breakthrough results. The gold standard for unbiased evaluation is a blind, prospective benchmark, in which different methods are evaluated on a newly generated test set that will only be disclosed after the results have been announced. A popular example in drug discovery is CASP (Critical Assessment of Structure Prediction) 2 , which enabled a revolution in protein structure prediction by systematically identifying valuable innovations in the community 3 . However, data acquisition in drug discovery is expensive and time-consuming, limiting the accessibility and availability of blind benchmarks to the general research community.

Big data and benchmarking initiatives to bridge the gap from AlphaFold to drug design

ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Tasks, Models, and Metrics.

AI-Based Protein Structure Prediction in Drug Discovery: Impacts and Challenges

Regularly updated benchmark sets for statistically correct evaluations of AlphaFold applications

Benchmarking Refined and Unrefined AlphaFold2 Structures for Hit Discovery

Artificial intelligence alphafold model for molecular biology and drug discovery: a machine-learning-driven informatics investigation

Review of AlphaFold 3: Transformative Advances in Drug Design and Therapeutics

Structure-based, deep-learning models for protein-ligand binding affinity prediction

The power and pitfalls of AlphaFold2 for structure prediction beyond rigid globular proteins

IMPROVING ALPHAFOLD2 PERFORMANCE WITH A GLOBAL METAGENOMIC & BIOLOGICAL DATA SUPPLY CHAIN

A call for an industry-led initiative to critically assess machine learning for real-world drug discovery

Benchmarking AlphaFold‐enabled molecular docking predictions for antibiotic discovery

Benchmarking AlphaFold for protein complex modeling reveals accuracy determinants

AlphaDesign: A graph protein design method and benchmark on AlphaFoldDB

AlphaFold2 in biomedical research: facilitating the development of diagnostic strategies for disease

APACE: AlphaFold2 and advanced computing as a service for accelerated discovery in biophysics

How good are AlphaFold models for docking-based virtual screening?

Advances in AI for Protein Structure Prediction: Implications for Cancer Drug Discovery and Development

Structure-based Drug Design Benchmark: Do 3D Methods Really Dominate?

An outlook on structural biology after AlphaFold: tools, limits and perspectives

PDBench: Evaluating Computational Methods for Protein Sequence Design