Abstract:Objectives: Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF). Target audience: Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices. Scope: We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.

LLM-IE: A Python Package for Generative Information Extraction with Large Language Models

Large Language Models for Generative Information Extraction: A Survey

IEPile: Unearthing Large Scale Schema-Conditioned Information Extraction Corpus

LLM-AIx: An open source pipeline for Information Extraction from unstructured medical text based on privacy pre-serving Large Language Models

LLM-AIx: An open source pipeline for Information Extraction from unstructured medical text based on privacy preserving Large Language Models

Large language models for biomedicine: foundations, opportunities, challenges, and best practices

An Empirical Study on Information Extraction using Large Language Models

LLMs in Biomedicine: A study on clinical Named Entity Recognition

Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction

Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?

Benchmarking Large Language Models in Evidence-Based Medicine

High-throughput Biomedical Relation Extraction for Semi-Structured Web Articles Empowered by Large Language Models

IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpus

Supervised Knowledge Makes Large Language Models Better In-context Learners

LEXI: Large Language Models Experimentation Interface

EHR Interaction Between Patients and AI: NoteAid EHR Interaction

Large Language Models for Scientific Information Extraction: An Empirical Study for Virology

A Platform for the Biomedical Application of Large Language Models