Abstract:Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and more importantly, they have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science,and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

Predicting Future Kinetic States of Physicochemical Systems Using Generative Pre-trained Transformer

Dynamic Molecular Graph-based Implementation for Biophysical Properties Prediction

Predicting Enzymatic Reactions with a Molecular Transformer

Exploring the conformational ensembles of protein-protein complex with transformer-based generative model

Exploring Time-Series Transformers for Spatio-Temporal Prediction of Microstructural Evolution of Polycrystalline Grain

Transformer Performance for Chemical Reactions: Analysis of Different Predictive and Evaluation Scenarios

Learning stochastic dynamics and predicting emergent behavior using transformers

3D-Transformer: Molecular Representation with Transformer in 3D Space

Transformer-based deep learning for predicting protein properties in the life sciences

Application of Transformers in Cheminformatics

KnoMol: A Knowledge-Enhanced Graph Transformer for Molecular Property Prediction

GraphGPT: A Graph Enhanced Generative Pretrained Transformer for Conditioned Molecular Generation

Transformers for molecular property prediction: Lessons learned from the past five years

MolGPT: Molecular Generation Using a Transformer-Decoder Model

KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction

Predicting Real-time Scientific Experiments Using Transformer models and Reinforcement Learning

Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction

Pre-training Transformers for Molecular Property Prediction Using Reaction Prediction

Regression Transformer: Concurrent sequence regression and generation for molecular language modeling

cMolGPT: A Conditional Generative Pre-Trained Transformer for Target-Specific De Novo Molecular Generation

G-Transformer: Counterfactual Outcome Prediction under Dynamic and Time-varying Treatment Regimes