ChemDFM: A Large Language Foundation Model for Chemistry

Zihan Zhao,Da Ma,Lu Chen,Liangtai Sun,Zihao Li,Yi Xia,Bo Chen,Hongshen Xu,Zichen Zhu,Su Zhu,Shuai Fan,Guodong Shen,Kai Yu,Xin Chen

2024-09-20

Abstract:Artificial intelligence (AI) has played an increasingly important role in chemical research. However, most models currently used in chemistry are specialist models that require training and tuning for specific tasks. A more generic and efficient solution would be an AI model that could address many tasks and support free-form dialogue in the broad field of chemistry. In its utmost form, such a generalist AI chemist could be referred to as Chemical General Intelligence. Large language models (LLMs) have recently logged tremendous success in the general domain of natural language processing, showing emerging task generalization and free-form dialogue capabilities. However, domain knowledge of chemistry is largely missing when training general-domain LLMs. The lack of such knowledge greatly hinders the performance of generalist LLMs in the field of chemistry. To this end, we develop ChemDFM, a pioneering LLM for chemistry trained on 34B tokens from chemical literature and textbooks, and fine-tuned using 2.7M instructions. As a result, it can understand and reason with chemical knowledge in free-form dialogue. Quantitative evaluations show that ChemDFM significantly surpasses most representative open-source LLMs. It outperforms GPT-4 on a great portion of chemical tasks, despite the substantial size difference. We have open-sourced the inference codes, evaluation datasets, and model weights of ChemDFM on Huggingface (<a class="link-external link-https" href="https://huggingface.co/AI4Chem/ChemLLM-7B-Chat" rel="external noopener nofollow">this https URL</a>).

Computation and Language,Digital Libraries

What problem does this paper attempt to address?

The paper aims to address the limitations of existing large language models (LLMs) in the field of chemistry when handling specialized chemical tasks. Specifically, most AI models used in chemical research are specialized models trained and fine-tuned for specific tasks, which leads to poor performance when these models face tasks in non-specific scenarios. To solve this problem, the paper proposes a new model called ChemDFM, a dialogue foundation model designed specifically for the field of chemistry.

ChemDFM acquires specialized knowledge in the field of chemistry through the following two stages of training:
1. **Domain Pre-training**: The model is pre-trained using a dataset of chemical literature and textbooks containing 3.4 billion tokens, enabling it to acquire basic chemical knowledge.
2. **Instruction Fine-tuning**: The model is further fine-tuned using a dataset containing 2.7 million chemical instructions, allowing it to understand and handle chemical language and molecular representations.

Ultimately, ChemDFM not only performs excellently on multiple chemical tasks but also surpasses large general-purpose language models like GPT-4 in some tasks. This indicates that ChemDFM possesses strong capabilities in handling chemical tasks and understanding natural language, making it a valuable tool or collaborator in chemical research.