Abstract:Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and more importantly, they have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science,and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

MOL-AE: Auto-Encoder Based Molecular Representation Learning With 3D Cloze Test Objective

An Image-enhanced Molecular Graph Representation Learning Framework

3D-Mol: A Novel Contrastive Learning Framework for Molecular Property Prediction with 3D Information

3D-Transformer: Molecular Representation with Transformer in 3D Space

MGMAE: Molecular Representation Learning by Reconstructing Heterogeneous Graphs with A High Mask Ratio

MolCloze - A Unified Cloze-style Self-supervised Molecular Structure Learning Model for Chemical Property Prediction.

Improving Chemical Autoencoder Latent Space and Molecular De Novo Generation Diversity with Heteroencoders

Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusion

3D-MolT5: Towards Unified 3D Molecule-Text Modeling with 3D Molecular Tokenization

MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter

MolX: Enhancing Large Language Models for Molecular Learning with A Multi-Modal Extension

Towards 3D Molecule-Text Interpretation in Language Models

GTAM: A Molecular Pretraining Model with Geometric Triangle Awareness

MolLM : a unified language model for integrating biomedical text with 2D and 3D molecular representations

Multi-Type Point Cloud Autoencoder: A Complete Equivariant Embedding for Molecule Conformation and Pose

3DMol-Net: Learn 3D Molecular Representation using Adaptive Graph Convolutional Network Based on Rotation Invariance

GeoMFormer: A General Architecture for Geometric Molecular Representation Learning

Sculpting Molecules in Text-3D Space: A Flexible Substructure Aware Framework for Text-Oriented Molecular Optimization

MoleMCL: a multi-level contrastive learning framework for molecular pre-training