Abstract:Proteins perform critical processes in all living systems: converting solar energy into chemical energy, replicating DNA, as the basis of highly performant materials, sensing and much more. While an incredible range of functionality has been sampled in nature, it accounts for a tiny fraction of the possible protein universe. If we could tap into this pool of unexplored protein structures, we could search for novel proteins with useful properties that we could apply to tackle the environmental and medical challenges facing humanity. This is the purpose of protein design. Sequence design is an important aspect of protein design, and many successful methods to do this have been developed. Recently, deep-learning methods that frame it as a classification problem have emerged as a powerful approach. Beyond their reported improvement in performance, their primary advantage over physics-based methods is that the computational burden is shifted from the user to the developers, thereby increasing accessibility to the design method. Despite this trend, the tools for assessment and comparison of such models remain quite generic. The goal of this paper is to both address the timely problem of evaluation and to shine a spotlight, within the Machine Learning community, on specific assessment criteria that will accelerate impact. We present a carefully curated benchmark set of proteins and propose a number of standard tests to assess the performance of deep learning based methods. Our robust benchmark provides biological insight into the behaviour of design methods, which is essential for evaluating their performance and utility. We compare five existing models with two novel models for sequence prediction. Finally, we test the designs produced by these models with AlphaFold2, a state-of-the-art structure-prediction algorithm, to determine if they are likely to fold into the intended 3D shapes.

The Protein Engineering Tournament: An Open Science Benchmark for Protein Modeling and Design

Results of the Protein Engineering Tournament: An Open Science Benchmark for Protein Modeling and Design

ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Tasks, Models, and Metrics.

PDBench: Evaluating Computational Methods for Protein Sequence Design

ProteinBench: A Holistic Evaluation of Protein Foundation Models

ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction

Evaluating representation learning on the protein structure universe

A Suite of Designed Protein Cages Using Machine Learning Algorithms and Protein Fragment-Based Protocols

A suite of designed protein cages using machine learning and protein fragment-based protocols

Computational protein design with evolutionary-based and physics-inspired modeling: current and future synergies

Critical Assessment of Protein Engineering (CAPE): A Student Challenge on the Cloud

FLOP: Tasks for Fitness Landscapes Of Protein wildtypes

Computational scoring and experimental evaluation of enzymes generated by neural networks

Robust deep learning based protein sequence design using ProteinMPNN

Illuminating protein space with a programmable generative model

Protein design on computers. Five new proteins: Shpilka, Grendel, Fingerclasp, Leather, and Aida

Computational Protein Design with Data‐driven Approaches: Recent Developments and Perspectives

ProteusAI: An Open-Source and User-Friendly Platform for Machine Learning-Guided Protein Design and Engineering

Low-N protein engineering with data-efficient deep learning

Computational Protein Design - Where It Goes?

Protein design: from computer models to artificial intelligence