Abstract:In academic research, systematic literature reviews are foundational and highly relevant, yet tedious to create due to the high volume of publications and labor-intensive processes involved. Systematic selection of relevant papers through conventional means like keyword-based filtering techniques can sometimes be inadequate, plagued by semantic ambiguities and inconsistent terminology, which can lead to sub-optimal outcomes. To mitigate the required extensive manual filtering, we explore and evaluate the potential of using Large Language Models (LLMs) to enhance the efficiency, speed, and precision of literature review filtering, reducing the amount of manual screening required. By using models as classification agents acting on a structured database only, we prevent common problems inherent in LLMs, such as hallucinations. We evaluate the real-world performance of such a setup during the construction of a recent literature survey paper with initially more than 8.3k potentially relevant articles under consideration and compare this with human performance on the same dataset. Our findings indicate that employing advanced LLMs like GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Flash, or Llama3 with simple prompting can significantly reduce the time required for literature filtering - from usually weeks of manual research to only a few minutes. Simultaneously, we crucially show that false negatives can indeed be controlled through a consensus scheme, achieving recalls >98.8% at or even beyond the typical human error threshold, thereby also providing for more accurate and relevant articles selected. Our research not only demonstrates a substantial improvement in the methodology of literature reviews but also sets the stage for further integration and extensive future applications of responsible AI in academic research practices.

Systematic literature reviews: Four applications for interdisciplinary research

Reviewing the animal literature: how to describe and choose between different types of literature reviews

Artificial intelligence and the conduct of literature reviews

Computational Literature Reviews: Method, Algorithms, and Roadmap

Literature reviews as independent studies: guidelines for academic practice

The Automation of Science? Possibilities and Boundaries of AI Applications for Conducting Systematic Literature Reviews

On the pragmatic design of literature studies in software engineering: an experience-based guideline

A systematic exploration of scoping and mapping literature reviews

Evaluating an interdisciplinary research project: Lessons learned for organisations, researchers and funders

A Systematic Literature Review of User Evaluation in Immersive Analytics

Advancing knowledge through literature reviews: ‘what’, ‘why’, and ‘how to contribute’

Applying meta-analysis to research on bilingualism: An introduction

Artificial intelligence for literature reviews: opportunities and challenges

A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence

Paving the Way for Mature Secondary Research: The Seven Types of Literature Review

Search and Selection Procedures of Literature Reviews in Behavior Analysis

Automatic Boolean Query Formulation for Systematic Review Literature Search

Cutting Through the Clutter: The Potential of LLMs for Efficient Filtration in Systematic Literature Reviews

Semantic City Planning Systems (SCPS): A Literature Review

Literature reviews in operations research: A new taxonomy and a meta review