Abstract:Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and more importantly, they have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science,and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

3D-TRANS: 3D Hierarchical Transformer for Shape Correspondence Learning.

SEFormer: Structure Embedding Transformer for 3D Object Detection

Hierarchical Shape-Consistent Transformer for Unsupervised Point Cloud Shape Correspondence

3D-Transformer: Molecular Representation with Transformer in 3D Space

LATFormer: Locality-Aware Point-View Fusion Transformer for 3D shape recognition

Spatial Transformer for 3D Point Clouds

A Transformer-Based Capsule Network for 3D Part–Whole Relationship Learning

Deep Implicit Templates for 3D Shape Representation

3D Former: Monocular Scene Reconstruction with 3D SDF Transformers

DeFormer: Integrating Transformers with Deformable Models for 3D Shape Abstraction from a Single Image

Stratified Transformer for 3D Point Cloud Segmentation

Geometry-aware 3D pose transfer using transformer autoencoder

Shape-Former: Bridging CNN and Transformer via ShapeConv for multimodal image matching

S$^3$-MonoDETR: Supervised Shape&Scale-perceptive Deformable Transformer for Monocular 3D Object Detection

Structure-Aware Cross-Modal Transformer for Depth Completion

3D-C2FT: Coarse-to-fine Transformer for Multi-view 3D Reconstruction

Hierarchical Spatial Transformer Network

DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation

OcTr: Octree-based Transformer for 3D Object Detection

Curriculumformer: Taming Curriculum Pre-Training for Enhanced 3-D Point Cloud Understanding