Regression with Large Language Models for Materials and Molecular Property Prediction

Ryan Jacobs,Maciej P. Polak,Lane E. Schultz,Hamed Mahdavi,Vasant Honavar,Dane Morgan

2024-09-10

Abstract:We demonstrate the ability of large language models (LLMs) to perform material and molecular property regression tasks, a significant deviation from the conventional LLM use case. We benchmark the Large Language Model Meta AI (LLaMA) 3 on several molecular properties in the QM9 dataset and 24 materials properties. Only composition-based input strings are used as the model input and we fine tune on only the generative loss. We broadly find that LLaMA 3, when fine-tuned using the SMILES representation of molecules, provides useful regression results which can rival standard materials property prediction models like random forest or fully connected neural networks on the QM9 dataset. Not surprisingly, LLaMA 3 errors are 5-10x higher than those of the state-of-the-art models that were trained using far more granular representation of molecules (e.g., atom types and their coordinates) for the same task. Interestingly, LLaMA 3 provides improved predictions compared to GPT-3.5 and GPT-4o. This work highlights the versatility of LLMs, suggesting that LLM-like generative models can potentially transcend their traditional applications to tackle complex physical phenomena, thus paving the way for future research and applications in chemistry, materials science and other scientific domains.

Materials Science,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the following question: Can large language models (LLMs) perform regression tasks for chemical and material properties directly from text prompts? Specifically, the authors explore the application of LLMs in predicting material and molecular properties, aiming to validate the following three questions:

1. **Can LLMs be trained to perform regression tasks for material and molecular properties from text prompts?**
2. **How does the performance of such LLMs compare to other established property models (e.g., state-of-the-art models or models based on standard regression methods)?**
3. **How is the regression performance of LLMs affected by the type of LLM (e.g., LLaMA 3 vs. GPT-3.5) and the input feature patterns used for fine-tuning the LLM (e.g., using SMILES strings, InChI strings, or atomic types and coordinates)?**

Through these questions, the authors hope to assess the potential application value of LLMs in the field of material and molecular science, particularly in reducing the need for domain-specific knowledge and optimizing machine learning models.