Abstract:How to obtain informative representations of molecules is a crucial prerequisite in AI-driven drug design and discovery. Recent researches abstract molecules as graphs and employ Graph Neural Networks (GNNs) for molecular representation learning. Nevertheless, two issues impede the usage of GNNs in real scenarios: (1) insufficient labeled molecules for supervised training; (2) poor generalization capability to new-synthesized molecules. To address them both, we propose a novel framework, GROVER, which stands for Graph Representation frOm self-superVised mEssage passing tRansformer. With carefully designed self-supervised tasks in node-, edge- and graph-level, GROVER can learn rich structural and semantic information of molecules from enormous unlabelled molecular data. Rather, to encode such complex information, GROVER integrates Message Passing Networks into the Transformer-style architecture to deliver a class of more expressive encoders of molecules. The flexibility of GROVER allows it to be trained efficiently on large-scale molecular dataset without requiring any supervision, thus being immunized to the two issues mentioned above. We pre-train GROVER with 100 million parameters on 10 million unlabelled molecules—the biggest GNN and the largest training dataset in molecular representation learning. We then leverage the pre-trained GROVER for molecular property prediction followed by task-specific fine-tuning, where we observe a huge improvement (more than 6% on average) from current state-of-the-art methods on 11 challenging benchmarks. The insights we gained are that well-designed self-supervision losses and largely-expressive pre-trained models enjoy the significant potential on performance boosting.

GraphGPT: A Graph Enhanced Generative Pretrained Transformer for Conditioned Molecular Generation

Graph Generative Pre-trained Transformer

cMolGPT: A Conditional Generative Pre-Trained Transformer for Target-Specific De Novo Molecular Generation

PrefixMol: Target- and Chemistry-aware Molecule Design Via Prefix Embedding

MolGPT: Molecular Generation Using a Transformer-Decoder Model

GPT like transformer based conditional molecule generator and a high drug-likeness (QED) dataset generation

RM-GPT: Enhance the comprehensive generative ability of molecular GPT model via LocalRNN and RealFormer

GraphGPT: Graph Learning with Generative Pre-trained Transformers

Adapt-cMolGPT: A Conditional Generative Pre-Trained Transformer with Adapter-Based Fine-Tuning for Target-Specific Molecular Generation

Pretraining Graph Transformer for Molecular Representation with Fusion of Multimodal Information

KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction

LigGPT: Molecular Generation using a Transformer-Decoder Model

Probabilistic generative transformer language models for generative design of molecules

Describe Molecules by a Heterogeneous Graph Neural Network with Transformer-like Attention for Supervised Property Predictions

Transformer Graph Variational Autoencoder for Generative Molecular Design

A knowledge-guided pre-training framework for improving molecular representation learning

GraphXForm: Graph transformer for computer-aided molecular design with application to extraction

A Graph VAE and Graph Transformer Approach to Generating Molecular Graphs

Self-Supervised Graph Transformer on Large-Scale Molecular Data

Molecular generative Graph Neural Networks for Drug Discovery

Molecular Graph Enhanced Transformer for Retrosynthesis Prediction