Abstract:Abstract Language models have shown the capacity to learn complex molecular distributions. In the field of molecular generation, they are designed to explore the distribution of molecules, and previous studies have demonstrated their ability to learn molecule sequences. In the early times, recurrent neural networks (RNNs) were widely used for feature extraction from sequence data and have been used for various molecule generation tasks. In recent years, the attention mechanism for sequence data has become popular. It captures the underlying relationships between words and is widely applied to language models. The Transformer-Layer, a model based on a self-attentive mechanism, also shines the same as the RNN-based model. In this research, we investigated the difference between RNNs and the Transformer-Layer to learn a more complex distribution of molecules. For this purpose, we experimented with three different generative tasks: the distributions of molecules with elevated scores of penalized LogP, multimodal distributions of molecules and the largest molecules in PubChem. We evaluated the models on molecular properties, basic metrics, Tanimoto similarity, etc. In addition, we applied two different representations of the molecule, SMILES and SELFIES. The results show that the two language models can learn complex molecular distributions and SMILES-based representation has better performance than SELFIES. The choice between RNNs and the Transformer-Layer needs to be based on the characteristics of dataset. RNNs work better on data focus on local features and decreases with multidistribution data, while the Transformer-Layer is more suitable when meeting molecular with larger weights and focusing on global features.

Empirical Evidence for the Fragment level Understanding on Drug Molecular Structure of LLMs

Fragment and Geometry Aware Tokenization of Molecules for Structure-Based Drug Design Using Language Models

Large-scale chemical language representations capture molecular structure and properties

Unlocking comprehensive molecular design across all scenarios with large language model and unordered chemical language

Language models in molecular discovery

Melting point prediction of organic molecules by deciphering the chemical structure into a natural language

Large language model for molecular chemistry

Infusing Linguistic Knowledge of SMILES into Chemical Language Models

t-SMILES: a fragment-based molecular representation framework for de novo ligand design

Molecular fragmentation as a crucial step in the AI-based drug development pathway

Hybrid fragment-SMILES tokenization for ADMET prediction in drug discovery

Pre-trained Molecular Language Models with Random Functional Group Masking

DrugLLM: Open Large Language Model for Few-shot Molecule Generation

MolX: Enhancing Large Language Models for Molecular Learning with A Multi-Modal Extension

Chemical Language Models for Molecular Design

Large Language Models as Molecular Design Engines

FraHMT: A Fragment-Oriented Heterogeneous Graph Molecular Generation Model for Target Proteins

FragLlama: Next-fragment Prediction for Molecular Design

SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery

Molecular language models: RNNs or transformer?

Can Large Language Models Understand Molecules?