Abstract:Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and more importantly, they have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science,and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

TransFoxMol: predicting molecular property with focused attention

3D-Transformer: Molecular Representation with Transformer in 3D Space

ComABAN: refining molecular representation with the graph attention mechanism to accelerate drug discovery

KnoMol: A Knowledge-Enhanced Graph Transformer for Molecular Property Prediction

TranGRU: Focusing on Both the Local and Global Information of Molecules for Molecular Property Prediction.

One Transformer Can Understand Both 2D & 3D Molecular Data

Synergistic Fusion of Graph and Transformer Features for Enhanced Molecular Property Prediction

Text-Guided Multi-Property Molecular Optimization with a Diffusion Language Model

MolTrans: Molecular Interaction Transformer for Drug Target Interaction Prediction

Molecular Representation Contrastive Learning Via Transformer Embedding to Graph Neural Networks

3DSGIMD: an Accurate and Interpretable Molecular Property Prediction Method Using 3D Spatial Graph Focusing Network and Structure-Based Feature Fusion

ABT-MPNN: an atom-bond transformer-based message-passing neural network for molecular property prediction

Relative molecule self-attention transformer

Few-shot learning with transformers via graph embeddings for molecular property prediction

TransScore: a Graph Model for Pose Scoring and Affinity Prediction Based on Transformer Convolution Network

Transformer-based molecular optimization beyond matched molecular pairs

DIG-Mol: A Contrastive Dual-Interaction Graph Neural Network for Molecular Property Prediction

AttentionMGT-DTA: A multi-modal drug-target affinity prediction using graph transformer and attention mechanism

3D-Mol: A Novel Contrastive Learning Framework for Molecular Property Prediction with 3D Information

Pharmacophoric-constrained heterogeneous graph transformer model for molecular property prediction