Abstract:Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and more importantly, they have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science,and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

Scalable Transformer for PDE Surrogate Modeling

FlowFormer: 3D Scene Flow Estimation for Point Clouds with Transformers

Multi-scale Time-stepping of Partial Differential Equations with Transformers

Transolver: A Fast Transformer Solver for PDEs on General Geometries

Exploring Efficient Partial Differential Equation Solution Using Speed Galerkin Transformer

SDPT: Semantic-Aware Dimension-Pooling Transformer for Image Segmentation

3D-Transformer: Molecular Representation with Transformer in 3D Space

Provable In-Context Learning of Linear Systems and Linear Elliptic PDEs with Transformers

Implicit factorized transformer approach to fast prediction of turbulent channel flows

Efficiently Scaling Transformer Inference

Unisolver: PDE-Conditional Transformers Are Universal PDE Solvers

A Unified Framework for Interpretable Transformers Using PDEs and Information Theory

Physics Informed Token Transformer for Solving Partial Differential Equations

Transformer-Powered Surrogates Close the ICF Simulation-Experiment Gap with Extremely Limited Data

Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity

Choose a Transformer: Fourier or Galerkin

RPPformer-Flow: Relative Position Guided Point Transformer for Scene Flow Estimation

InParformer: Evolutionary Decomposition Transformers with Interactive Parallel Attention for Long-Term Time Series Forecasting

A 28nm 49.7TOPS/W Sparse Transformer Processor with Random-Projection-Based Speculation, Multi-Stationary Dataflow, and Redundant Partial Product Elimination

Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning

Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEs