Abstract:Recent advancements have integrated Language Models (LMs) into a drug discovery pipeline. However, existing models mostly work with SMILES and SELFIES chemical string representations, which lack spatial features vital for drug discovery. Additionally, attempts to translate chemical 3D structures into text format encounter issues such as excessive length and insufficient atom connectivity information. To address these issues, we introduce nach0-pc, a model combining domain-specific encoder and textual representation to handle spatial arrangement of atoms effectively. Our approach utilizes a molecular point cloud encoder for concise and order-invariant structure representation. We introduce a novel pre-training scheme for molecular point clouds to distillate the knowledge from spatial molecular structures datasets. After fine-tuning within both single-task and multi-task frameworks, nach0-pc demonstrates performance comparable with other diffusion models in terms of generated samples quality across several established spatial molecular generation tasks. Notably, our model is a multi-task approach, in contrast to diffusion models being limited to single tasks. Additionally, it is capable of processing point cloud-related data, which language models are not capable of handling due to memory limitations. These lead to our model having reduced training and inference time while maintaining on par performance.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on the challenges encountered by existing language models when dealing with 3D molecular structures in drug discovery. Specifically, the paper points out: 1. **Most existing models rely on chemical string representations such as SMILES and SELFIES**: Although these representation methods can capture the structural information of molecular graphs, they lack the spatial features that are crucial for drug discovery. For example, they cannot accurately describe the spatial arrangement and interaction of atoms. 2. **Methods that attempt to convert chemical 3D structures into text formats have flaws**: The text formats (such as PDB, CIF, and XYZ) generated by these methods are usually too long, requiring dozens of tokens to represent each atom, and lack information on atomic connectivity. In addition, these formats need to rely on external software tools when reconstructing chemical bonds, and these tools are very sensitive to small errors in atomic positions, which may lead to incorrect reconstruction of chemical graphs or the breakage of molecules. To solve these problems, the authors introduced the nach0 - pc model, which is a multi - task language model that combines domain - specific encoders and text representations. The main contributions of this model include: 1. **Proposing a novel nach0 - pc model**: By integrating specialized molecular point - cloud encoders and tokens, the standard encoder - decoder language model is enhanced. This model can effectively handle the input and output of 3D molecular structures. 2. **Introducing a new pre - training method**: By performing dropout operations on entire sub - fragments, the model is trained to predict the missing parts of incomplete molecular point clouds. The incomplete molecular point clouds generated by this method are generated through fragment omission and/or blurring strategies. 3. **Demonstrating the performance of the model through extensive experiments**: On six spatial molecular generation tasks, the performance of the nach0 - pc model is better than or comparable to that of the baseline language model and the state - of - the - art diffusion model. In conclusion, this paper aims to improve the efficiency and accuracy of language models in handling 3D molecular structures by introducing the nach0 - pc model, thereby better supporting the drug discovery process.

nach0-pc: Multi-task Language Model with Molecular Point Cloud Encoder

nach0: Multimodal Natural and Chemical Languages Foundation Model

MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter

MolX: Enhancing Large Language Models for Molecular Learning with A Multi-Modal Extension

MolLM : a unified language model for integrating biomedical text with 2D and 3D molecular representations

Improving Molecular Properties Prediction Through Latent Space Fusion

Multilingual Molecular Representation Learning via Contrastive Pre-training

Towards 3D Molecule-Text Interpretation in Language Models

ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area

MolFusion: Multimodal Fusion Learning for Molecular Representations via Multi-granularity Views

3D-MolT5: Towards Unified 3D Molecule-Text Modeling with 3D Molecular Tokenization

InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery

MolBind: Multimodal Alignment of Language, Molecules, and Proteins

MolCloze - A Unified Cloze-style Self-supervised Molecular Structure Learning Model for Chemical Property Prediction.

3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation

MolMVC: Enhancing molecular representations for drug-related tasks through multi-view contrastive learning

Chemical Language Model Linker: blending text and molecules with modular adapters

MAMMAL -- Molecular Aligned Multi-Modal Architecture and Language

Interactive Molecular Discovery with Natural Language

Beyond Chemical Language: A Multimodal Approach to Enhance Molecular Property Prediction