Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

Fabio Dennstädt,Johannes Zink,Paul Martin Putora,Janna Hastings,Nikola Cihoric
DOI: https://doi.org/10.1186/s13643-024-02575-4
2024-06-17
Systematic Reviews
Abstract:Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose.
medicine, general & internal
What problem does this paper attempt to address?
The paper aims to explore how to utilize large language models (LLMs) for title and abstract screening in literature reviews to automate the assessment of the relevance of publications in the biomedical field. Specifically, the study designs an LLM-based method to automatically evaluate the relevance of literature titles and abstracts and defines which literature should be included in systematic literature reviews by setting different thresholds. The researchers tested four different publicly available large-scale language models and conducted experiments on ten published biomedical literature datasets as well as a newly created dataset. - **Main Objective**: Develop a flexible method to automate the screening of literature titles and abstracts using large language models, thereby reducing the time researchers need to screen relevant literature during systematic literature reviews. - **Specific Problem**: The current process of systematic literature reviews is time-consuming and cumbersome, especially during the title and abstract screening stage. The study aims to improve the efficiency and consistency of this process by introducing LLMs. - **Method Innovation**: Proposes a new strategy by constructing structured prompts to guide LLMs in evaluating the relevance of literature and defining classifiers based on the scoring results to distinguish between relevant and irrelevant literature. - **Potential Value**: If this method can be effectively implemented, it will significantly shorten the time required to complete systematic literature reviews and improve the consistency and reproducibility of the literature screening process. In summary, the study attempts to address the time-consuming and inefficient manual screening of titles and abstracts in systematic literature reviews and explores the potential application of LLMs in this process.