Abstract:Proteins perform critical processes in all living systems: converting solar energy into chemical energy, replicating DNA, as the basis of highly performant materials, sensing and much more. While an incredible range of functionality has been sampled in nature, it accounts for a tiny fraction of the possible protein universe. If we could tap into this pool of unexplored protein structures, we could search for novel proteins with useful properties that we could apply to tackle the environmental and medical challenges facing humanity. This is the purpose of protein design. Sequence design is an important aspect of protein design, and many successful methods to do this have been developed. Recently, deep-learning methods that frame it as a classification problem have emerged as a powerful approach. Beyond their reported improvement in performance, their primary advantage over physics-based methods is that the computational burden is shifted from the user to the developers, thereby increasing accessibility to the design method. Despite this trend, the tools for assessment and comparison of such models remain quite generic. The goal of this paper is to both address the timely problem of evaluation and to shine a spotlight, within the Machine Learning community, on specific assessment criteria that will accelerate impact. We present a carefully curated benchmark set of proteins and propose a number of standard tests to assess the performance of deep learning based methods. Our robust benchmark provides biological insight into the behaviour of design methods, which is essential for evaluating their performance and utility. We compare five existing models with two novel models for sequence prediction. Finally, we test the designs produced by these models with AlphaFold2, a state-of-the-art structure-prediction algorithm, to determine if they are likely to fold into the intended 3D shapes.

Regularly updated benchmark sets for statistically correct evaluations of AlphaFold applications

ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Tasks, Models, and Metrics.

Big data and benchmarking initiatives to bridge the gap from AlphaFold to drug design

Benchmarking of AlphaFold2 accuracy self-estimates as indicators of empirical model quality and ranking: a comparison with independent model quality assessment programmes

AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models

Benchmarking Refined and Unrefined AlphaFold2 Structures for Hit Discovery

AlphaFold2 for protein structure prediction: Best practices and critical analyses

Highly significant improvement of protein sequence alignments with AlphaFold2

PDBench: Evaluating Computational Methods for Protein Sequence Design

AlphaFold Protein Structure Database in 2024: providing structure coverage for over 214 million protein sequences

The power and pitfalls of AlphaFold2 for structure prediction beyond rigid globular proteins

Highly accurate protein structure prediction with AlphaFold

AlphaFold two years on: validation and impact

Applying and improving AlphaFold at CASP14

Accurate structure prediction of biomolecular interactions with AlphaFold 3

AlphaFold 2: Why It Works and Its Implications for Understanding the Relationships of Protein Sequence, Structure, and Function

IMPROVING ALPHAFOLD2 PERFORMANCE WITH A GLOBAL METAGENOMIC & BIOLOGICAL DATA SUPPLY CHAIN

High-throughput prediction of protein conformational distributions with subsampled AlphaFold2

Ins and outs of AlphaFold2 transmembrane protein structure predictions