Extracting accurate materials data from research papers with conversational language models and prompt engineering

Maciej P. Polak,Dane Morgan

DOI: https://doi.org/10.1038/s41467-024-45914-8

IF: 16.6

2024-02-21

Nature Communications

Abstract:Abstract There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work, we propose the method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data’s correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data, we find precision and recall both close to 90% from the best conversational LLMs, like GPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to , due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using .

multidisciplinary sciences

What problem does this paper attempt to address?

This paper proposes a solution to automatically and accurately extract material data from research papers. Traditional methods for automated data extraction rely on natural language processing and language models, requiring extensive upfront work, domain expertise, and coding. The paper introduces a new method called ChatExtract, which utilizes advanced conversational large language models to enable nearly fully automated high-precision data extraction with minimal initial effort and no specialized background.

ChatExtract applies carefully designed prompts to conversational language models to identify sentences containing data, extract the data, and ensure its correctness through a series of follow-up questions, overcoming potential inaccuracies that large language models may provide. This approach is applicable to any conversational language model, with testing showing precision and recall rates close to 90% for material data extraction, particularly demonstrating excellent performance on GPT-4.

Key features of ChatExtract include information retention, redundancy design, and introducing uncertainty to improve accuracy. Through this approach, efficient data extraction can be achieved even without additional training. The paper also presents examples of using ChatExtract to construct databases for critical cooling rates of metallic glasses and yield strength of high-entropy alloys.

In conclusion, this paper addresses the challenges of automating the extraction of material data from research papers and presents a simple, transferable, and accurate method that holds promise as a powerful tool for future data extraction.