Abstract:Accurate prediction of pharmacological properties of small molecules is becoming increasingly important in drug discovery. Traditional feature-engineering approaches heavily rely on handcrafted descriptors and/or fingerprints, which need extensive human expert knowledge. With the rapid progress of artificial intelligence technology, data-driven deep learning methods have shown unparalleled advantages over feature-engineering-based methods. However, existing deep learning methods usually suffer from the scarcity of labeled data and the inability to share information between different tasks when applied to predicting molecular properties, thus resulting in poor generalization capability. Here, we proposed a novel multitask learning BERT (Bidirectional Encoder Representations from Transformer) framework, named MTL-BERT, which leverages large-scale pre-training, multitask learning, and SMILES (simplified molecular input line entry specification) enumeration to alleviate the data scarcity problem. MTL-BERT first exploits a large amount of unlabeled data through self-supervised pretraining to mine the rich contextual information in SMILES strings and then fine-tunes the pretrained model for multiple downstream tasks simultaneously by leveraging their shared information. Meanwhile, SMILES enumeration is used as a data enhancement strategy during the pretraining, fine-tuning, and test phases to substantially increase data diversity and help to learn the key relevant patterns from complex SMILES strings. The experimental results showed that the pretrained MTL-BERT model with few additional fine-tuning can achieve much better performance than the state-of-the-art methods on most of the 60 practical molecular datasets. Additionally, the MTL-BERT model leverages attention mechanisms to focus on SMILES character features essential to target properties for model interpretability.

Extracting Predictive Representations from Hundreds of Millions of Molecules

A Systematic Survey of Chemical Pre-trained Models

Toward Robust Self-Training Paradigm for Molecular Prediction Tasks

Supervised Pretraining for Molecular Force Fields and Properties Prediction

From molecules to scaffolds to functional groups: building context-dependent molecular representation via multi-channel learning

InstructBio: A Large-scale Semi-supervised Learning Paradigm for Biochemical Problems

Tetrahedral Molecular Pretraining for Enhanced Property Prediction

Masked Molecule Modeling: A New Paradigm of Molecular Representation Learning for Chemistry Understanding

An effective self-supervised framework for learning expressive molecular global representations to drug discovery

Learn molecular representations from large-scale unlabeled molecules for drug discovery

Inductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT

Relative molecule self-attention transformer

Review of Unsupervised Pretraining Strategies for Molecules Representation

Self-Supervised Molecular Representation Learning With Topology and Geometry

Triple Generative Self-Supervised Learning Method for Molecular Property Prediction

MG-BERT: leveraging unsupervised atomic representation learning for molecular property prediction

Pushing the Boundaries of Molecular Property Prediction for Drug Discovery with Multitask Learning BERT Enhanced by SMILES Enumeration

Pre-training Protein Models with Molecular Dynamics Simulations for Drug Binding

ChemBERTa-2: Towards Chemical Foundation Models

3D Molecular Pretraining via Localized Geometric Generation

A Knowledge-Driven Self-Supervised Approach for Molecular Generation