Abstract:Spatial structures in the 3D space are important to determine molecular properties. Recent papers use geometric deep learning to represent molecules and predict properties. These papers, however, are computationally expensive in capturing long-range dependencies of input atoms; and more importantly, they have not considered the non-uniformity of interatomic distances, thus failing to learn context-dependent representations at different scales. To deal with such issues, we introduce 3D-Transformer, a variant of the Transformer for molecular representations that incorporates 3D spatial information. 3D-Transformer operates on a fully-connected graph with direct connections between atoms. To cope with the non-uniformity of interatomic distances, we develop a multi-scale self-attention module that exploits local fine-grained patterns with increasing contextual scales. As molecules of different sizes rely on different kinds of spatial features, we design an adaptive position encoding module that adopts different position encoding methods for small and large molecules. Finally, to attain the molecular representation from atom embeddings, we propose an attentive farthest point sampling algorithm that selects a portion of atoms with the assistance of attention scores, overcoming handicaps of the virtual node and previous distance-dominant downsampling methods. We validate 3D-Transformer across three important scientific domains: quantum chemistry, material science,and proteomics. Our experiments show significant improvements over state-of-the-art models on the crystal property prediction task and the protein-ligand binding affinity prediction task, and show better or competitive performance in quantum chemistry molecular datasets. This work provides clear evidence that biochemical tasks can gain consistent benefits from 3D molecular representations and different tasks require different position encoding methods.

Molecular representation learning with language models and domain-relevant auxiliary tasks

An Image-enhanced Molecular Graph Representation Learning Framework

Pushing the Boundaries of Molecular Property Prediction for Drug Discovery with Multitask Learning BERT Enhanced by SMILES Enumeration

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

MG-BERT: leveraging unsupervised atomic representation learning for molecular property prediction

Molecular Graph Representation Learning Integrating Large Language Models with Domain-specific Small Models

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Large-scale chemical language representations capture molecular structure and properties

Transfer Learning across Different Chemical Domains: Virtual Screening of Organic Materials with Deep Learning Models Pretrained on Small Molecule and Chemical Reaction Data

Multilingual Molecular Representation Learning via Contrastive Pre-training

Self-Supervised Molecular Representation Learning With Topology and Geometry

Language models in molecular discovery

3D-Transformer: Molecular Representation with Transformer in 3D Space

BERT Learns (and Teaches) Chemistry

Knowledge-based BERT: a Method to Extract Molecular Features Like Computational Chemists

Knowledge-based BERT: a method to extract molecular features such as computational chemists

Transformers and Large Language Models for Chemistry and Drug Discovery

A merged molecular representation learning for molecular properties prediction with a web-based service

Enhancing Molecular Property Prediction through Task-Oriented Transfer Learning: Integrating Universal Structural Insights and Domain-Specific Knowledge

SELFormer: Molecular Representation Learning via SELFIES Language Models