Abstract:Objective: This study aims to summarize the usage of Large Language Models (LLMs) in the process of creating a scientific review. We look at the range of stages in a review that can be automated and assess the current state-of-the-art research projects in the field. Materials and Methods: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar databases by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on which uses OpenAI gpt-4o model. ChatGPT was used to clean extracted data and generate code for figures in this manuscript, ChatGPT and <a class="link-external link-http" href="http://Scite.ai" rel="external noopener nofollow">this http URL</a> were used in drafting all components of the manuscript, except the methods and discussion sections. Results: 3,788 articles were retrieved, and 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n=126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n=26, 15.1%) were actual reviews that used LLM during their creation. Most citations focused on automation of a particular stage of review, such as Searching for publications (n=60, 34.9%), and Data extraction (n=54, 31.4%). When comparing pooled performance of GPT-based and BERT-based models, the former were better in data extraction with mean precision 83.0% (SD=10.4), and recall 86.0% (SD=9.8), while being slightly less accurate in title and abstract screening stage (Maccuracy=77.3%, SD=13.0). Discussion/Conclusion: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. The results looked promising, and we anticipate that LLMs will change in the near future the way the scientific reviews are conducted.

A Large Language Model-Powered Literature Review for HighAngle Annular Dark Field Imaging

Towards Efficient Large Language Models for Scientific Text: A Review

Materials science in the era of large language models: a perspective

An Interdisciplinary Outlook on Large Language Models for Scientific Research

A Bibliometric Review of Large Language Models Research from 2017 to 2023

A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery

Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT

Scientific Large Language Models: A Survey on Biological & Chemical Domains

Large Language Models for Data Annotation: A Survey

A Survey for Large Language Models in Biomedicine

From Text to Insight: Large Language Models for Materials Science Data Extraction

Exploring Large Language Models for Feature Selection: A Data-centric Perspective

Automated Review Generation Method Based on Large Language Models

Large Language Models for Data Annotation and Synthesis: A Survey

Large Language Models for Generative Information Extraction: A Survey

Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review

Evaluating Large Language Models for Radiology Natural Language Processing

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

Can Large Language Models Learn the Physics of Metamaterials? An Empirical Study with ChatGPT

The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review