Abstract:Proteins perform critical processes in all living systems: converting solar energy into chemical energy, replicating DNA, as the basis of highly performant materials, sensing and much more. While an incredible range of functionality has been sampled in nature, it accounts for a tiny fraction of the possible protein universe. If we could tap into this pool of unexplored protein structures, we could search for novel proteins with useful properties that we could apply to tackle the environmental and medical challenges facing humanity. This is the purpose of protein design. Sequence design is an important aspect of protein design, and many successful methods to do this have been developed. Recently, deep-learning methods that frame it as a classification problem have emerged as a powerful approach. Beyond their reported improvement in performance, their primary advantage over physics-based methods is that the computational burden is shifted from the user to the developers, thereby increasing accessibility to the design method. Despite this trend, the tools for assessment and comparison of such models remain quite generic. The goal of this paper is to both address the timely problem of evaluation and to shine a spotlight, within the Machine Learning community, on specific assessment criteria that will accelerate impact. We present a carefully curated benchmark set of proteins and propose a number of standard tests to assess the performance of deep learning based methods. Our robust benchmark provides biological insight into the behaviour of design methods, which is essential for evaluating their performance and utility. We compare five existing models with two novel models for sequence prediction. Finally, we test the designs produced by these models with AlphaFold2, a state-of-the-art structure-prediction algorithm, to determine if they are likely to fold into the intended 3D shapes.

Datasets for Benchmarking RNA Design Algorithms

RnaBench: A Comprehensive Library for RNA Modelling

RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design

RNA-NRD: a non-redundant RNA structural dataset for benchmarking and functional analysis

RNA3DB: A structurally-dissimilar dataset split for training and benchmarking deep learning models for RNA structure prediction

Toward Increasing the Credibility of RNA Design

PDBench: Evaluating Computational Methods for Protein Sequence Design

Benchmarking RNA all-atom force fields using hairpin motifs

Sequence Design for RNA-RNA Interactions

Detecting Ribosnitches with RNA Folding Algorithms: a Genome-Wide Benchmark

Learning to Design RNA

Sequence Design Using RNAstructure

State-of-the-RNArt: benchmarking current methods for RNA 3D structure prediction

Machine learning for RNA 2D structure prediction benchmarked on experimental data

Fixed-Parameter Tractable Sampling for RNA Design with Multiple Target Structures

Computational strategies for the automated design of RNA nanoscale structures from building blocks using NanoTiler

PDB-Struct: A Comprehensive Benchmark for Structure-based Protein Design

Scalable and Interpretable Identification of Minimal Undesignable RNA Structure Motifs with Rotational Invariance

Algorithmic Design of 3D Wireframe RNA Polyhedra

RNAsolo: a repository of cleaned PDB-derived RNA 3D structures

Simulated Annealing for RNA Design with SIMARD