Abstract:Background: Large language models (LLMs) have significantly enhanced the Natural Language Processing (NLP), offering significant potential in facilitating medical literature review. However, the accuracy, stability and prompt strategies associated with LLMs in extracting complex medical information have not been adequately investigated. Our study assessed the capabilities of GPT-3.5 and GPT-4.0 in extracting or summarizing seven crucial medical information items from the title and abstract of research papers. We also validated the impact of prompt engineering strategies and the effectiveness of evaluating metrics. Methodology: We adopted a stratified sampling method to select 100 papers from the teaching schools and departments in the LKS Faculty of Medicine, University of Hong Kong, published between 2015 and 2023. GPT-3.5 and GPT-4.0 were instructed to extract seven pieces of information, including study design, sample size, data source, patient, intervention, comparison, and outcomes. The experiment incorporated three prompt engineering strategies: persona, chain-of-thought and few-shot prompting. We employed three metrics to assess the alignment between the GPT output and the ground truth: BERTScore, ROUGE-1 and a self-developed GPT-4.0 evaluator. Finally, we evaluated and compared the proportion of correct answers among different GPT versions and prompt engineering strategies. Results: GPT demonstrated robust capabilities in accurately extracting medical information from titles and abstracts. The average accuracy of GPT-4.0, when paired with the optimal prompt engineering strategy, ranged from 0.688 to 0.964 among the seven items, with sample size achieving the highest score and intervention yielding the lowest. GPT version was shown to be a statistically significant factor in model performance, but prompt engineering strategies did not exhibit cumulative effects on model performance. Additionally, our results showed that the GPT-4.0 evaluator outperformed the ROUGE-1 and BERTScore in assessing the alignment of information (Accuracy: GPT-4.0 Evaluator: 0.9714, ROUGE-1: 0.9429, BERTScore: 0.8714). Conclusion: Our result confirms the effectiveness of LLMs in extracting medical information, suggesting their potential as efficient tools for literature review. We recommend utilizing an advanced version of LLMs to enhance the model performance, while prompt engineering strategies should be tailored to the specific tasks. Additionally, LLMs show promise as an evaluation tool to assess the model performance related to complex information processing.

Exploring the Potential of Large Language Models in Molecular Tasks: An Insightful Evaluation with GPT‐4

The Future of Molecular Studies Through the Lens of Large Language Models.

The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4

An Evaluation of Large Language Models in Bioinformatics Research

MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction

GPT-4 as an interface between researchers and computational software: improving usability and reproducibility

Benchmarking Large Language Models for Molecule Prediction Tasks

Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective

Can Large Language Models Empower Molecular Property Prediction?

Evaluating the Potential of Leading Large Language Models in Reasoning Biology Questions

What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks

Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT

3D Molecular Pocket-based Generation with Token-only Large Language Model

Evaluation of Open-Source Large Language Models for Metal-Organic Frameworks Research

GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction

Can Large Language Models Understand Molecules?

Large Language Model in Medical Information Extraction from Titles and Abstracts with Prompt Engineering Strategies: A Comparative Study of GPT-3.5 and GPT-4

Fine-tuning Large Language Models for Chemical Text Mining

Exploring the Potential of Large Language Models in Graph Generation

Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations

Large property models: a new generative machine-learning formulation for molecules