A Foundation Model for Chemical Design and Property Prediction

Feiyang Cai,Tianyu Zhu,Tzuen-Rong Tzeng,Yongping Duan,Ling Liu,Srikanth Pilla,Gang Li,Feng Luo

2024-10-29

Abstract:Artificial intelligence (AI) has significantly advanced computational chemistry research. However, traditional AI methods often rely on task-specific model designs and training, which constrain both the scalability of model size and generalization across different tasks. Here, we introduce ChemFM, a large-scale foundation model specifically developed for chemistry, comprising up to 3 billion parameters. ChemFM is pre-trained on 178 million molecules using self-supervised causal language modeling to extract generalizable molecular representations. This model can be adapted to diverse downstream chemical applications using both full-parameter and parameter-efficient fine-tuning methods. ChemFM consistently outperforms state-of-the-art approaches across multiple chemical tasks. Notably, it achieves up to 67.48% performance improvement across 34 property prediction benchmarks, up to 33.31% reduction in mean average deviation between conditioned and actual properties of generated molecules in conditional molecular generation tasks, and up to 3.7% top-1 accuracy improvement across 4 reaction prediction datasets. Moreover, ChemFM demonstrates superior performance in predicting antibiotic activity and cytotoxicity, highlighting its potential to advance the discovery of novel antibiotics. We anticipate that ChemFM will significantly advance chemistry research by providing a foundation model capable of effectively generalizing across a broad range of tasks with minimal additional training.

Computational Engineering, Finance, and Science

What problem does this paper attempt to address?

The paper attempts to address several key challenges in chemical design and property prediction:

1. **High data annotation cost**: Traditional artificial intelligence methods in the field of chemistry usually require a large amount of high-quality data for training. The acquisition of such data often requires a significant amount of time and funding, as they typically rely on complex laboratory experiments.

2. **Poor model generalization**: Existing task-specific models, although performing well on certain specific tasks, often struggle to capture the general patterns of molecular internal structure dependencies and contextual relationships. This leads to severe overfitting and poor prediction performance on newly emerging molecular features.

3. **Limited model scalability**: Traditional chemical models face limitations in scaling, which affects their generality and scalability across different tasks.

To address the above issues, the paper introduces a large-scale foundational model named ChemFM, specifically designed for the field of chemistry, containing up to 3 billion parameters. By pre-training with self-supervised causal language modeling on 178 million molecules, ChemFM can extract molecular representations with broad applicability. Additionally, ChemFM can adapt to various downstream chemical applications through full-parameter or parameter-efficient fine-tuning methods, thereby demonstrating performance that surpasses existing state-of-the-art methods in multiple chemical tasks, particularly in molecular property prediction, conditional molecular generation, and reaction prediction.