Abstract:Large Language Models (LLMs) stand at the forefront of a number of Natural Language Processing (NLP) tasks. Despite the widespread adoption of LLMs in NLP, much of their potential in broader fields remains largely unexplored, and significant limitations persist in their design and implementation. Notably, LLMs struggle with structured data, such as graphs, and often falter when tasked with answering domain-specific questions requiring deep expertise, such as those in biology and chemistry. In this paper, we explore a fundamental question: Can LLMs effectively handle molecule prediction tasks? Rather than pursuing top-tier performance, our goal is to assess how LLMs can contribute to diverse molecule tasks. We identify several classification and regression prediction tasks across six standard molecule datasets. Subsequently, we carefully design a set of prompts to query LLMs on these tasks and compare their performance with existing Machine Learning (ML) models, which include text-based models and those specifically designed for analysing the geometric structure of molecules. Our investigation reveals several key insights: Firstly, LLMs generally lag behind ML models in achieving competitive performance on molecule tasks, particularly when compared to models adept at capturing the geometric structure of molecules, highlighting the constrained ability of LLMs to comprehend graph data. Secondly, LLMs show promise in enhancing the performance of ML models when used collaboratively. Lastly, we engage in a discourse regarding the challenges and promising avenues to harness LLMs for molecule prediction tasks. The code and models are available at

Can Large Language Models Understand Molecules?

Large language model for molecular chemistry

Large Language Models as Molecular Design Engines

Exploring the Potential of Large Language Models in Molecular Tasks: An Insightful Evaluation with GPT‐4

The Future of Molecular Studies Through the Lens of Large Language Models.

Large-scale chemical language representations capture molecular structure and properties

What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks

Are large language models superhuman chemists?

From Words to Molecules: A Survey of Large Language Models in Chemistry

Can Large Language Models Empower Molecular Property Prediction?

Benchmarking Large Language Models for Molecule Prediction Tasks

MolX: Enhancing Large Language Models for Molecular Learning with A Multi-Modal Extension

Large Language Models are In-Context Molecule Learners

GIT-Mol: A multi-modal large language model for molecular science with graph, image, and text

SmileyLlama: Modifying Large Language Models for Directed Chemical Space Exploration

Large Language Models for Biomolecular Analysis: from Methods to Applications

Structured Chemistry Reasoning with Large Language Models

DrugLLM: Open Large Language Model for Few-shot Molecule Generation

Large language models for science and medicine

An Evaluation of Large Language Models in Bioinformatics Research