Abstract:Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and more importantly, they have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science,and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

MTS-Net: an Enriched Topology-Aware Architecture for Molecular Graph Representation Learning

Pretraining Graph Transformer for Molecular Representation with Fusion of Multimodal Information

3D-Transformer: Molecular Representation with Transformer in 3D Space

Molecular Graph Enhanced Transformer for Retrosynthesis Prediction

3DMol-Net: Learn 3D Molecular Representation using Adaptive Graph Convolutional Network Based on Rotation Invariance

Describe Molecules by a Heterogeneous Graph Neural Network with Transformer-like Attention for Supervised Property Predictions

Autoencoding Undirected Molecular Graphs With Neural Networks

Molecular Graph Transformer: Stepping Beyond ALIGNN Into Long-Range Interactions

One Transformer Can Understand Both 2D & 3D Molecular Data

Molecular CT: Unifying Geometry and Representation Learning for Molecules at Different Scales

GeoT: A Geometry-aware Transformer for Reliable Molecular Property Prediction and Chemically Interpretable Representation Learning

CTAGE: Curvature-Based Topology-Aware Graph Embedding for Learning Molecular Representations

TransG-net: transformer and graph neural network based multi-modal data fusion network for molecular properties prediction

Molecular substructure graph attention network for molecular property identification in drug discovery

Molecular Joint Representation Learning via Multi-modal Information of SMILES and Graphs

Adaptive Multi-Neighborhood Attention based Transformer for Graph Representation Learning

Topology-guided Hypergraph Transformer Network: Unveiling Structural Insights for Improved Representation

Predicting Molecule-Target Interaction by Learning Biomedical Network and Molecule Representations

Molecular Topological Profile (MOLTOP) -- Simple and Strong Baseline for Molecular Graph Classification

Molecular Graph Representation Learning via Structural Similarity Information

KnoMol: A Knowledge-Enhanced Graph Transformer for Molecular Property Prediction