Diffusion on language model embeddings for protein sequence generation

Viacheslav Meshchaninov,Pavel Strashnov,Andrey Shevtsov,Fedor Nikolaev,Nikita Ivanisenko,Olga Kardymon,Dmitry Vetrov

2024-03-06

Abstract:Protein design requires a deep understanding of the inherent complexities of the protein universe. While many efforts lean towards conditional generation or focus on specific families of proteins, the foundational task of unconditional generation remains underexplored and undervalued. Here, we explore this pivotal domain, introducing DiMA, a model that leverages continuous diffusion on embeddings derived from the protein language model, ESM-2, to generate amino acid sequences. DiMA surpasses leading solutions, including autoregressive transformer-based and discrete diffusion models, and we quantitatively illustrate the impact of the design choices that lead to its superior performance. We extensively evaluate the quality, diversity, distribution similarity, and biological relevance of the generated sequences using multiple metrics across various modalities. Our approach consistently produces novel, diverse protein sequences that accurately reflect the inherent structural and functional diversity of the protein space. This work advances the field of protein design and sets the stage for conditional models by providing a robust framework for scalable and high-quality protein sequence generation.

Machine Learning,Artificial Intelligence,Biomolecules

What problem does this paper attempt to address?

The paper focuses on the problem of unconditional generation of protein sequences, which is a fundamental but under-explored challenge in the field of protein design. Current methods often focus on conditional generation or specific protein families, neglecting comprehensive unconditional generation. The paper proposes a model called DiMA, which utilizes the embedding and continuous diffusion methods of protein language model (ESM-2) to generate amino acid sequences. DiMA surpasses existing solutions, including autoregressive Transformer models and discrete diffusion models, in terms of quality and diversity, and demonstrates the advantages of its design choices through multiple evaluation metrics. The working principle of DiMA is to first encode the amino acid sequence into continuous representations using ESM-2, and then train a diffusion model to reconstruct the disrupted embeddings. During the inference stage, protein embeddings are generated by iteratively refining from random Gaussian embeddings and decoded into amino acid sequences. The generated sequences exhibit high quality, diversity, and accurately reflect the structural and functional diversity of proteins. The paper also conducts extensive evaluations, including sequence quality, diversity, distribution similarity, and biological relevance, emphasizing that the protein sequences generated by DiMA can capture the characteristics of the training data, providing a powerful framework for the field of protein design and laying the foundation for conditional models. Additionally, the paper explores the impact of different design choices and training strategies.

Diffusion on language model embeddings for protein sequence generation

A Unified Conditional Diffusion Framework for Dual Protein Targets Based Bioactive Molecule Generation

Diffusion Language Models Are Versatile Protein Learners

PRO-LDM: Protein Sequence Generation with a Conditional Latent Diffusion Model

AMP-Diffusion: Integrating Latent Diffusion with Protein Language Models for Antimicrobial Peptide Generation

Protein generation with evolutionary diffusion: sequence is all you need

Protein A-like Peptide Design Based on Diffusion and ESM2 Models

Generative models for protein sequence modeling: recent advances and future directions

DPLM-2: A Multimodal Diffusion Protein Language Model

MeMDLM: De Novo Membrane Protein Design with Masked Discrete Diffusion Protein Language Models

Importance Weighted Expectation-Maximization for Protein Sequence Design

Generating Novel, Designable, and Diverse Protein Structures by Equivariantly Diffusing Oriented Residue Clouds

Graph Denoising Diffusion for Inverse Protein Folding

A Latent Diffusion Model for Protein Structure Generation

Atom-by-atom protein generation and beyond with language models

Structure-based protein and small molecule generation using EGNN and diffusion models: A comprehensive review

Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models

Secondary Structure-Guided Novel Protein Sequence Generation with Latent Graph Diffusion

Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design

Protein Design by Directed Evolution Guided by Large Language Models

Evolutionary context-integrated deep sequence modeling for protein engineering