Complementary multi-modality molecular self-supervised learning via non-overlapping masking for property prediction

Ao Shen,Mingzhi Yuan,Yingfan Ma,Jie Du,Manning Wang

DOI: https://doi.org/10.1093/bib/bbae256

IF: 9.5

2024-05-29

Briefings in Bioinformatics

Abstract:Self-supervised learning plays an important role in molecular representation learning because labeled molecular data are usually limited in many tasks, such as chemical property prediction and virtual screening. However, most existing molecular pre-training methods focus on one modality of molecular data, and the complementary information of two important modalities, SMILES and graph, is not fully explored. In this study, we propose an effective multi-modality self-supervised learning framework for molecular SMILES and graph. Specifically, SMILES data and graph data are first tokenized so that they can be processed by a unified Transformer-based backbone network, which is trained by a masked reconstruction strategy. In addition, we introduce a specialized non-overlapping masking strategy to encourage fine-grained interaction between these two modalities. Experimental results show that our framework achieves state-of-the-art performance in a series of molecular property prediction tasks, and a detailed ablation study demonstrates efficacy of the multi-modality framework and the masking strategy.

biochemical research methods,mathematical & computational biology

What problem does this paper attempt to address?

This paper mainly discusses how to use complementary multimodal molecular self-supervised learning to predict chemical properties. Existing molecular pre-training methods mostly focus on one modality of data, while this paper proposes a multimodal self-supervised learning framework for SMILES sequences and molecular graphs. Through a non-overlapping masking strategy, this framework promotes fine-grained interaction between the two modalities to improve prediction performance. Experimental results show that this method achieves state-of-the-art performance in a range of molecular property prediction tasks, and the efficacy of the multimodal framework and masking strategy is verified through ablation studies. The paper points out that SMILES and graph modalities provide complementary information and their combined use can enhance molecular representation performance. Therefore, they design a non-overlapping masking strategy, which ensures that the masked parts in SMILES and graph do not overlap during pre-training, thereby encouraging the model to use direct information from the other modality to reconstruct the masked parts, enhancing the interaction between modalities.