Abstract:This study is dedicated to assessing the capabilities of large language models (LLMs) such as GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo in extracting structured information from scientific documents in materials science. To this end, we primarily focus on two critical tasks of information extraction: (i) a named entity recognition (NER) of studied materials and physical properties and (ii) a relation extraction (RE) between these entities. Due to the evident lack of datasets within Materials Informatics (MI), we evaluated using SuperMat, based on superconductor research, and MeasEval, a generic measurement evaluation corpus. The performance of LLMs in executing these tasks is benchmarked against traditional models based on the BERT architecture and rule-based approaches (baseline). We introduce a novel methodology for the comparative analysis of intricate material expressions, emphasising the standardisation of chemical formulas to tackle the complexities inherent in materials science information assessment. For NER, LLMs fail to outperform the baseline with zero-shot prompting and exhibit only limited improvement with few-shot prompting. However, a GPT-3.5-Turbo fine-tuned with the appropriate strategy for RE outperforms all models, including the baseline. Without any fine-tuning, GPT-4 and GPT-4-Turbo display remarkable reasoning and relationship extraction capabilities after being provided with merely a couple of examples, surpassing the baseline. Overall, the results suggest that although LLMs demonstrate relevant reasoning skills in connecting concepts, specialised models are currently a better choice for tasks requiring extracting complex domain-specific entities like materials. These insights provide initial guidance applicable to other materials science sub-domains in future work.

Annotating Materials Science Text: A Semi-Automated Approach for Crafting Outputs with Gemini Pro

Towards Development of Automated Knowledge Maps and Databases for Materials Engineering using Large Language Models

Materials Synthesis Insights from Scientific Literature via Text Extraction and Machine Learning

From Text to Insight: Large Language Models for Materials Science Data Extraction

The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures

Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models

Materials science in the era of large language models: a perspective

Automatically Extracting Action Graphs from Materials Science Synthesis Procedures

Synergizing Human Expertise and AI Efficiency with Language Model for Microscopy Operation and Automated Experiment Design

NLP meets Materials Science: Quantifying the presentation of materials data in scientific literature

Fine-tuning Large Language Models for Chemical Text Mining

Mining experimental data from Materials Science literature with Large Language Models: an evaluation study

Automated electrosynthesis reaction mining with multimodal large language models (MLLMs)

Polymetis:Large Language Modeling for Multiple Material Domains

Semi-supervised machine-learning classification of materials synthesis procedures

Are LLMs Ready for Real-World Materials Discovery?

Structured information extraction from complex scientific text with fine-tuned large language models

Reconstructing Materials Tetrahedron: Challenges in Materials Information Extraction

Extracting accurate materials data from research papers with conversational language models and prompt engineering