Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering

Maciej P. Polak,Dane Morgan

DOI: https://doi.org/10.1038/s41467-024-45914-8

2024-02-21

Abstract:There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.

Computation and Language,Materials Science

What problem does this paper attempt to address?

The paper primarily addresses the problem of how to automate and accurately extract material data from research papers. Traditional automation methods require a significant amount of preliminary work, domain knowledge, and coding. The paper proposes the ChatExtract method, which leverages advanced conversational language models (such as ChatGPT) and prompt engineering to achieve high-precision data extraction with almost zero initial effort and background. By designing a series of prompts to identify sentences containing data, extract the data, and validate the correctness of the data through subsequent questioning, ChatExtract overcomes the issue of potentially inaccurate responses from large language models. ChatExtract can be applied to any conversational language model and achieves close to 90% precision and recall in material data testing. Research indicates that the excellent performance of this approach benefits from the information retention capability, meaningful redundancy, and introduction of uncertainty in dialogue models. The paper also demonstrates examples of using ChatExtract to build databases for critical cooling rates of metallic glasses and yield strength of high-entropy alloys, and suggests similar approaches have the potential to become powerful tools for future data extraction.