Abstract:Large language models (LLMs) bear promise as a fast and accurate material modeling paradigm for evaluation, analysis, and design. Their vast number of trainable parameters necessitates a wealth of data to achieve accuracy and mitigate overfitting. However, experimental measurements are often limited and costly to obtain in sufficient quantities for finetuning. To this end, we present a physics-based training pipeline that tackles the pathology of data scarcity. The core enabler is a physics-based modeling framework that generates a multitude of synthetic data to align the LLM to a physically consistent initial state before finetuning. Our framework features a two-phase training strategy: (1) utilizing the large-in-amount while less accurate synthetic data for supervised pretraining, and (2) finetuning the phase-1 model with limited experimental data. We empirically demonstrate that supervised pretraining is vital to obtaining accurate finetuned LLMs, via the lens of learning polymer flammability metrics where cone calorimeter data is sparse.

What problem does this paper attempt to address?

This paper mainly discusses how to effectively train large-scale language models (LLMs) to predict the properties of polymers under data scarcity. Currently, due to the large number of parameters in LLMs, a large amount of experimental data is required to ensure accuracy and prevent overfitting. However, experimental data is often limited and expensive. To solve this problem, the paper proposes a physics-based training pipeline, which includes two-stage training strategies: 1. Supervised pre-training stage: Pre-train the LLMs with a large amount of synthetic data (which may not be very accurate) generated by a physics-based modeling framework to ensure the initial state of LLMs is consistent with physics. 2. Stage 2 fine-tuning: Fine-tune the Stage 1 model with limited experimental data to reduce numerical errors and simplify the impact of assumptions, maximizing prediction accuracy. The paper also introduces a physics-based group contribution method to generate hypothetical polymers with various basic physical properties. These properties can be used to build physical simulation models or generate a large amount of synthetic data of interest. The research found that through this method, the prediction accuracy of LLMs can be significantly improved by more than 50%, even with extremely limited experimental data. The main contributions of the paper include: - Generating meaningful hypothetical polymer datasets for LLM pre-training using a physics-based group contribution method. - Proposing a physics-based modeling framework that uses group contributions as inputs to generate synthetic data. - Designing a two-stage prediction-correction training strategy, first pre-training with synthetic data and then fine-tuning with a small amount of experimental data. - Demonstrating a learning framework under data scarcity by combining LLMs, physics modeling, and experimental measurements, and validating the effectiveness of the framework using the example of polymer combustion characteristics. The experimental results of the paper show that this approach can significantly improve the accuracy of LLM predictions for polymer properties, especially in cases where experimental data is limited.

Large language models, physics-based modeling, experimental measurements: the trinity of data-scarce learning of polymer properties

Can Large Language Models Learn the Physics of Metamaterials? An Empirical Study with ChatGPT

Exploring large language models for microstructure evolution in materials

Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT

Large Language Models for Biomolecular Analysis: from Methods to Applications

Regression with Large Language Models for Materials and Molecular Property Prediction

LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset

LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

Large property models: a new generative machine-learning formulation for molecules

Large Language Models are Catalyzing Chemistry Education

Transferring a molecular foundation model for polymer property predictions

A Prompt-Engineered Large Language Model, Deep Learning Workflow for Materials Classification

Materials science in the era of large language models: a perspective

Gas permeability, diffusivity, and solubility in polymers: Simulation-experiment data fusion and multi-task machine learning

Large Property Models: A New Generative Paradigm for Molecules

Can Large Language Models Empower Molecular Property Prediction?

Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions

Polymetis:Large Language Modeling for Multiple Material Domains

Benchmarking Large Language Models for Molecule Prediction Tasks

Large Language Models for Material Property Predictions: elastic constant tensor prediction and materials design