Abstract:Accurate prediction of pharmacological properties of small molecules is becoming increasingly important in drug discovery. Traditional feature-engineering approaches heavily rely on handcrafted descriptors and/or fingerprints, which need extensive human expert knowledge. With the rapid progress of artificial intelligence technology, data-driven deep learning methods have shown unparalleled advantages over feature-engineering-based methods. However, existing deep learning methods usually suffer from the scarcity of labeled data and the inability to share information between different tasks when applied to predicting molecular properties, thus resulting in poor generalization capability. Here, we proposed a novel multitask learning BERT (Bidirectional Encoder Representations from Transformer) framework, named MTL-BERT, which leverages large-scale pre-training, multitask learning, and SMILES (simplified molecular input line entry specification) enumeration to alleviate the data scarcity problem. MTL-BERT first exploits a large amount of unlabeled data through self-supervised pretraining to mine the rich contextual information in SMILES strings and then fine-tunes the pretrained model for multiple downstream tasks simultaneously by leveraging their shared information. Meanwhile, SMILES enumeration is used as a data enhancement strategy during the pretraining, fine-tuning, and test phases to substantially increase data diversity and help to learn the key relevant patterns from complex SMILES strings. The experimental results showed that the pretrained MTL-BERT model with few additional fine-tuning can achieve much better performance than the state-of-the-art methods on most of the 60 practical molecular datasets. Additionally, the MTL-BERT model leverages attention mechanisms to focus on SMILES character features essential to target properties for model interpretability.

FG-BERT: a Generalized and Self-Supervised Functional Group-Based Molecular Representation Learning Framework for Properties Prediction.

MG-BERT: leveraging unsupervised atomic representation learning for molecular property prediction

SMG-BERT: integrating stereoscopic information and chemical representation for molecular property prediction

Knowledge-based BERT: a method to extract molecular features such as computational chemists

Knowledge-based BERT: a Method to Extract Molecular Features Like Computational Chemists

Pushing the Boundaries of Molecular Property Prediction for Drug Discovery with Multitask Learning BERT Enhanced by SMILES Enumeration

CIMG-BERT: Pretraining Bidirectional Transformers with Chemistry Knowledge for Molecular Property Prediction

A knowledge-guided pre-training framework for improving molecular representation learning

KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction

Ensemble Model With Bert,Roberta and Xlnet For Molecular property prediction

An improved multi-modal representation-learning model based on fusion networks for property prediction in drug discovery

A fingerprints based molecular property prediction method using the BERT model

GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction

Ligandformer: A Graph Neural Network for Predicting Compound Property with Robust Interpretation

Enhancing Molecular Property Prediction through Task-Oriented Transfer Learning: Integrating Universal Structural Insights and Domain-Specific Knowledge

MultiModal-Learning for Predicting Molecular Properties: A Framework Based on Image and Graph Structures

A deep learning framework for predicting molecular property based on multi-type features fusion

A merged molecular representation learning for molecular properties prediction with a web-based service

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Molecular representation learning with language models and domain-relevant auxiliary tasks

Integrating Chemical Language and Molecular Graph in Multimodal Fused Deep Learning for Drug Property Prediction