Abstract:Background and Aims Most clinical information is encoded as text, but extracting quantitative information from text is challenging. Large Language Models (LLMs) have emerged as powerful tools for natural language processing and can parse clinical text. However, many LLMs including ChatGPT reside in remote data centers, which disqualifies them from processing personal healthcare data. We present an open-source pipeline using the local LLM 'Llama 2' for extracting quantitative information from clinical text and evaluate its use to detect clinical features of decompensated liver cirrhosis. Methods We tasked the LLM to identify five key clinical features of decompensated liver cirrhosis in a zero- and one-shot way without any model training. Our specific objective was to identify abdominal pain, shortness of breath, confusion, liver cirrhosis, and ascites from 500 patient medical histories from the MIMIC IV dataset. We compared LLMs with three different sizes and a variety of pre-specified prompt engineering approaches. Model predictions were compared against the ground truth provided by the consent of three blinded medical experts. Results Our open-source pipeline yielded in highly accurate extraction of quantitative features from medical free text. Clinical features which were explicitly mentioned in the source text, such as liver cirrhosis and ascites, were detected with a sensitivity of 100% and 95% and a specificity of 96% and 95%, respectively from the 70 billion parameter model. Other clinical features, which are often paraphrased in a variety of ways, such as the presence of confusion, were detected only with a sensitivity of 76% and a specificity of 94%. Abdominal pain was detected with a sensitivity of 84% and a specificity of 97%. Shortness of breath was detected with a sensitivity of 87% and a specificity of 96%. The larger version of Llama 2 with 70b parameters outperformed the smaller version with 7b parameters in all tasks. Prompt engineering improved zero-shot performance, particularly for smaller model sizes. Conclusion Our study successfully demonstrates the capability of using locally deployed LLMs to extract clinical information from free text. The hardware requirements are so low that not only on-premise, but also point-of-care deployment of LLMs are possible.

Moving Beyond ChatGPT: Local Large Language Models (LLMs) and the Secure Analysis of Confidential Unstructured Text Data in Social Work Research

From Text to Tables: A Local Privacy Preserving Large Language Model for Structured Information Retrieval from Medical Documents

Psychological Assessments with Large Language Models: A Privacy-Focused and Cost-Effective Approach

Harnessing large language models' zero-shot and few-shot learning capabilities for regulatory research

Large Language Models: A New Approach for Privacy Policy Analysis at Scale

A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly

A Survey of Clinicians’ Views of the Utility of Large Language Models

Privacy-preserving large language models for structured medical information retrieval

Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review

LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions

Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?

Can LLMs be Fooled? Investigating Vulnerabilities in LLMs

Evaluating Large Language Models for Public Health Classification and Extraction Tasks

Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

Apprentices to Research Assistants: Advancing Research with Large Language Models

Inductive thematic analysis of healthcare qualitative interviews using open-source large language models: How does it compare to traditional methods?

Detection of suicidality from medical text using privacy-preserving large language models

Using LLMs to Aid Annotation and Collection of Clinically-Enriched Data in Bipolar Disorder and Schizophrenia

A survey of Large Language Model use in a hospital, research, and teaching campus

Large Language Models and Healthcare Alliance: Potential and Challenges of Two Representative Use Cases