Abstract:Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and more importantly, they have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science,and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

Hie-Transformer: A Hierarchical Hybrid Transformer For Abstractive Article Summarization

A study of extractive summarization of long documents incorporating local topic and hierarchical information

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

Hierarchical Multi-modal Transformer for Cross-modal Long Document Classification

Hierarchical Multi-modal Prompting Transformer for Multi-modal Long Document Classification

A Survey on Long Text Modeling with Transformers

HDT: Hierarchical Document Transformer

Revisiting Transformer-based Models for Long Document Classification

Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention

3D-Transformer: Molecular Representation with Transformer in 3D Space

ERNIE-Doc: A Retrospective Long-Document Modeling Transformer

HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval

Hierarchical Transformers for Long Document Classification

Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understanding

Metaformer: A Transformer That Tends to Mine Metaphorical-Level Information

Multi-modal long document classification based on Hierarchical Prompt and Multi-modal Transformer

Hierarchical Transformers Are More Efficient Language Models

Transformer-xl: Language modeling with longer-term dependency

A Transformer-Based Hierarchical Variational AutoEncoder Combined Hidden Markov Model for Long Text Generation