Abstract:Background: Accurate medical coding is essential for clinical and administrative purposes but complicated, time-consuming, and biased. This study compares Retrieval-Augmented Generation (RAG)-enhanced LLMs to provider-assigned codes in producing ICD-10-CM codes from emergency department (ED) clinical records. Methods: Retrospective cohort study using 500 ED visits randomly selected from the Mount Sinai Health System between January and April 2024. The RAG system integrated past 1,038,066 ED visits data (2021-2023) into the LLMs predictions to improve coding accuracy. Nine commercial and open-source LLMs were evaluated. The primary outcome was a head-to-head comparison of the ICD-10-CM codes generated by the RAG-enhanced LLMs and those assigned by the original providers. A panel of four physicians and two LLMs blindly reviewed the codes, comparing the RAG-enhanced LLM and provider-assigned codes on accuracy and specificity. Findings: RAG-enhanced LLMs demonstrated superior performance to provider coders in both the accuracy and specificity of code assignments. In a targeted evaluation of 200 cases where discrepancies existed between GPT-4 and provider-assigned codes, human reviewers favored GPT-4 for accuracy in 447 instances, compared to 277 instances where providers codes were preferred (p<0.001). Similarly, GPT-4 was selected for its superior specificity in 509 cases, whereas human coders were preferred in only 181 cases (p<0.001). Smaller open-access models, such as Llama-3.1-70B, also demonstrated substantial scalability when enhanced with RAG, with 218 instances of accuracy preference compared to 90 for providers' codes. Furthermore, across all models, the exact match rate between LLM-generated and provider-assigned codes significantly improved following RAG integration, with Qwen-2-7B increasing from 0.8% to 17.6% and Gemma-2-9b-it improving from 7.2% to 26.4%. Interpretation: RAG-enhanced LLMs improve medical coding accuracy in EDs, suggesting clinical workflow applications. These findings show that generative AI can improve clinical outcomes and reduce administrative burdens.

Exploring LLM Multi-Agents for ICD Coding

Modelling long medical documents and code associations for explainable automatic ICD coding

Automated clinical coding using off-the-shelf large language models

Benchmarking Large Language Models for Extraction of International Classification of Diseases Codes from Clinical Documentation

MMedAgent: Learning to Use Medical Tools with Multi-modal Agent

Towards Automated ICD Coding Using Deep Learning

TransICD: Transformer Based Code-wise Attention Model for Explainable ICD Coding

Assessing Retrieval-Augmented Large Language Model Performance in Emergency Department ICD-10-CM Coding Compared to Human Coders

Surpassing GPT-4 Medical Coding with a Two-Stage Approach

Towards Automatic ICD Coding Via Knowledge Enhanced Multi-Task Learning

MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making

MEDCO: Medical Education Copilots Based on A Multi-Agent Framework

Large language models enabled multiagent ensemble method for efficient EHR data labeling

AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

A Survey of Automated ICD Coding: Development, Challenges, and Applications

EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records

Multi-label Few-shot ICD Coding as Autoregressive Generation with Prompt

Automatic ICD-10-CM coding via Lambda-Scaled attention based deep learning model

Automated ICD coding for primary diagnosis via clinically interpretable machine learning

Large Language Model in Medical Informatics: Direct Classification and Enhanced Text Representations for Automatic ICD Coding

Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings