Abstract:Background: International Classification of Diseases codes are widely used to describe diagnosis information, but manual coding relies heavily on human interpretation, which can be expensive, time consuming, and prone to errors. With the transition from the International Classification of Diseases, Ninth Revision, to the International Classification of Diseases, Tenth Revision (ICD-10), the coding process has become more complex, highlighting the need for automated approaches to enhance coding efficiency and accuracy. Inaccurate coding can result in substantial financial losses for hospitals, and a precise assessment of outcomes generated by a natural language processing (NLP)-driven autocoding system thus assumes a critical role in safeguarding the accuracy of the Taiwan diagnosis related groups (Tw-DRGs). Objective: This study aims to evaluate the feasibility of applying an International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM), autocoding system that can automatically determine diagnoses and codes based on free-text discharge summaries to facilitate the assessment of Tw-DRGs, specifically principal diagnosis and major diagnostic categories (MDCs). Methods: By using the patient discharge summaries from Kaohsiung Medical University Chung-Ho Memorial Hospital (KMUCHH) from April 2019 to December 2020 as a reference data set we developed artificial intelligence (AI)-assisted ICD-10-CM coding systems based on deep learning models. We constructed a web-based user interface for the AI-assisted coding system and deployed the system to the workflow of the certified coding specialists (CCSs) of KMUCHH. The data used for the assessment of Tw-DRGs were manually curated by a CCS with the principal diagnosis and MDC was determined from discharge summaries collected at KMUCHH from February 2023 to April 2023. Results: Both the reference data set and real hospital data were used to assess performance in determining ICD-10-CM coding, principal diagnosis, and MDC for Tw-DRGs. Among all methods, the GPT-2 (OpenAI)-based model achieved the highest F1-score, 0.667 (F1-score 0.851 for the top 50 codes), on the KMUCHH test set and a slightly lower F1-score, 0.621, in real hospital data. Cohen κ evaluation for the agreement of MDC between the models and the CCS revealed that the overall average κ value for GPT-2 (κ=0.714) was approximately 12.2 percentage points higher than that of the hierarchy attention network (κ=0.592). GPT-2 demonstrated superior agreement with the CCS across 6 categories of MDC, with an average κ value of approximately 0.869 (SD 0.033), underscoring the effectiveness of the developed AI-assisted coding system in supporting the work of CCSs. Conclusions: An NLP-driven AI-assisted coding system can assist CCSs in ICD-10-CM coding by offering coding references via a user interface, demonstrating the potential to reduce the manual workload and expedite Tw-DRG assessment. Consistency in performance affirmed the effectiveness of the system in supporting CCSs in ICD-10-CM coding and the judgment of Tw-DRGs.

Validation of diagnostic codes of major clinical outcomes in a National Health Insurance database

Validity of the diagnosis of diabetic microvascular complications in Korean national health insurance claim data

Positive predictive value of ICD-10 codes for acute myocardial infarction in Japan: a validation study at a single center

Cardiovascular Research Using the Korean National Health Information Database

Validation of algorithms to identify stroke risk factors in patients with acute ischemic stroke, transient ischemic attack, or intracerebral hemorrhage in an administrative claims database

Development and validation of novel simple prognostic model for predicting mortality in Korean intensive care units using national insurance claims data

The influence of diagnostic subgroups, patient- and hospital characteristics for the validity of cardiovascular diagnoses–Data from a Norwegian hospital trust

Acute myocardial infarction: Development and application of an ICD-10-CM-based algorithm to a large U.S. healthcare claims-based database

Limitations of NHIC claim code-based surveillance and the necessity of UDI implementation in Korea

Abstract 75: Validation of Vascular Location Subcodes for Acute Ischemic Stroke by the International Classification of Diseases-10

Abstract WP77: Changes in Acute Stroke Management and Outcomes in Korea Between 2013-2014 and 2018

Evaluating a Natural Language Processing-Driven, AI-Assisted International Classification of Diseases, 10th Revision, Clinical Modification, Coding System for Diagnosis Related Groups in a Real Hospital Environment: Algorithm Development and Validation Study

Validation of I71.3 code for ruptured abdominal aortic aneurysm in Korea: misplaced diagnosis in claims data

Epidemiology and Functional Outcome of Acute Stroke Patients in Korea using Nationwide data

Validation of ICD‐10 hospital discharge diagnosis codes to identify incident and recurrent ischemic stroke from a US integrated healthcare system

Background and Data Configuration Process of a Nationwide Population-Based Study Using the Korean National Health Insurance System

The development and implementation of stroke risk prediction model in National Health Insurance Service's personal health record

Clinical Impact of Beta Blockers in Patients with Myocardial Infarction from the Korean National Health Insurance Database

Evaluating the concordance between International Classification of Diseases, Tenth Revision Code and stroke severity as measured by the National Institutes of Health Stroke Scale

Impact of the Dedicated Neurointensivists on the Outcome in Patients with Ischemic Stroke Based on the Linked Big Data for Stroke in Korea

Evaluating a Natural Language Processing–Driven, AI-Assisted International Classification of Diseases, 10th Revision, Clinical Modification, Coding System for Diagnosis Related Groups in a Real Hospital Environment: Algorithm Development and Validation Study