Abstract:Large Language Models (LLMs) with strong abilities in natural language processing tasks have emerged and have been applied in various kinds of areas such as science, finance and software engineering. However, the capability of LLMs to advance the field of chemistry remains unclear. In this paper, rather than pursuing state-of-the-art performance, we aim to evaluate capabilities of LLMs in a wide range of tasks across the chemistry domain. We identify three key chemistry-related capabilities including understanding, reasoning and explaining to explore in LLMs and establish a benchmark containing eight chemistry tasks. Our analysis draws on widely recognized datasets facilitating a broad exploration of the capacities of LLMs within the context of practical chemistry. Five LLMs (GPT-4, GPT-3.5, Davinci-003, Llama and Galactica) are evaluated for each chemistry task in zero-shot and few-shot in-context learning settings with carefully selected demonstration examples and specially crafted prompts. Our investigation found that GPT-4 outperformed other models and LLMs exhibit different competitive levels in eight chemistry tasks. In addition to the key findings from the comprehensive benchmark analysis, our work provides insights into the limitation of current LLMs and the impact of in-context learning settings on LLMs' performance across various chemistry tasks. The code and datasets used in this study are available at <a class="link-external link-https" href="https://github.com/ChemFoundationModels/ChemLLMBench" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper aims to explore the application capabilities of large language models (LLMs) in the field of chemistry and establish a comprehensive benchmark to evaluate the performance of these models in various chemical tasks. Specifically: 1. **Research Background and Objectives**: - Although large language models have demonstrated strong capabilities in fields such as natural language processing, finance, and software engineering, their application potential in the field of chemistry has not yet been fully explored. - The goal of this paper is not to pursue state-of-the-art performance but to evaluate the capabilities of LLMs through a series of chemical tasks. 2. **Main Contributions**: - Identified three key chemistry-related capabilities: understanding, reasoning, and interpretation, and based on this, established a benchmark comprising 8 chemical tasks. - Selected five LLMs (GPT-4, GPT-3.5, Davinci-003, Llama, and Galactica) for evaluation in zero-shot and few-shot contextual learning settings. - Found that GPT-4 performed best in most tasks and revealed differences in competitiveness among different models across various tasks. 3. **Specific Tasks**: - Includes 8 tasks: name prediction, property prediction, yield prediction, reaction prediction, retrosynthesis, text-based molecular design, molecular description generation, and reagent selection. 4. **Experimental Analysis**: - GPT-4 generally performed well but showed poor performance in tasks requiring precise understanding of molecular SMILES representation (e.g., reaction prediction and retrosynthesis). - In classification tasks (e.g., property prediction and yield prediction) and ranking tasks (e.g., reagent selection), GPT models achieved competitive results. - For text-based tasks (e.g., molecular design and description generation), GPT models performed well, although there were still deficiencies in precise matching. 5. **Further Discussion**: - The paper also discusses the capabilities of different LLMs, the impact of in-context learning (ICL), and the effect of temperature parameters on performance. - Results indicate that high-quality and numerous ICL examples are crucial for performance improvement, while the SELFIES representation did not show an advantage over SMILES. In summary, through systematic benchmarking, this paper not only demonstrates the current capabilities and limitations of LLMs in chemical tasks but also provides valuable insights for future research.

What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks

From Words to Molecules: A Survey of Large Language Models in Chemistry

Are large language models superhuman chemists?

LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset

Benchmarking Large Language Models for Molecule Prediction Tasks

ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models

Fine-tuning Large Language Models for Chemical Text Mining

Large Language Models are Catalyzing Chemistry Education

A Review of Large Language Models and Autonomous Agents in Chemistry

Structured Chemistry Reasoning with Large Language Models

ChemDFM: A Large Language Foundation Model for Chemistry

ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area

Assessment of chemistry knowledge in large language models that generate code

Exploring the Potential of Large Language Models in Molecular Tasks: An Insightful Evaluation with GPT‐4

Comparison of LLMs in Extracting Synthesis Conditions and Generating Q&A Datasets for Metal-Organic Frameworks

Augmenting large language models with chemistry tools

Leveraging large language models for predictive chemistry

Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry

Assessment of Fine-Tuned Large Language Models for Real-World Chemistry and Material Science Applications