Abstract:Mengfei Wang, 1 Jianyong Wei, 2 Yao Zeng, 3 Lisong Dai, 4 Bicong Yan, 4 Yueqi Zhu, 4 Xiaoer Wei, 4 Yidong Jin, 1 Yuehua Li 4 1 School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, People's Republic of China; 2 Clinical Research Center, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, People's Republic of China; 3 Shenyang University of Technology, Shenyang, Liaoning, People's Republic of China; 4 Institute of Diagnostic and Interventional Radiology, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, People's Republic of China Correspondence: Yuehua Li, Institute of Diagnostic and Interventional Radiology, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, No. 600 Yishan Road, Shanghai, 200233, People's Republic of China, Email Introduction: Mechanical thrombectomy (MTB) is a critical procedure for acute ischemic stroke (AIS) patients. However, the free-text format of MTB surgical records limits the formulation of effective postoperative patient management and rehabilitation plans. This study compares the efficacy of large language models (LLMs) in structuring data from these free-text MTB surgical record. Methods: This retrospective study collected a total of 382 MTB surgical records from a tertiary hospital. An initial analysis of 30 surgical record from these records provided a guiding prompt for LLMs, focusing on basic and advanced characteristics, such as occlusion locations, thrombectomy maneuvers, reperfusion status, and intraoperative complications. Six LLMs—ChatGPT, GPT-4, GeminiPro, ChatGLM4, Spark3, and QwenMax—were assessed against data extracted by neuroradiologists and a junior physician for comparison. The all 382 surgical records were used to test the performance of LLMs. The performance of the LLMs was quantified using Accuracy, Sensitivity, Specificity, AUC, and MSE as an additional metric for advanced characteristics. Results: All LLMs showed high performance in characteristic extraction, achieving an average accuracy of 95.09 ± 4.98% across 48 items, and 78.05 ± 4.2% overall. GLM4 and GPT-4 were most accurate in advanced characteristics extraction, with accuracies of 84.03% and 82.20%, respectively. The processing time for LLMs averaged 73.10 ± 10.86 seconds of six models, significantly faster than the 427.88 seconds for manual extraction by physicians. Conclusion: LLMs, particularly GLM4 and GPT-4, efficiently and accurately structured both general and advanced characteristics from MTB surgical record, outperforming manual extraction methods and demonstrating potential for enhancing clinical data management in AIS treatment. Keywords: large language models, free-text report, mechanical thrombectomy, acute ischemia stroke Acute Ischemic Stroke (AIS) stands as a leading cause of mortality and debilitating conditions among adults globally. 1 The burden of stroke has increased significantly over recent decades, with the Global Burden of Disease (GBD) study showing that the annual number of strokes and stroke-related deaths increased substantially from 1990 to 2019. 2 This trend is particularly evident in China, where the burden of stroke remains critically high. 3,4 The surgical records of mechanical thrombectomy (MTB) detail essential intraoperative findings, including the extent of blood flow restoration and reperfusion status. These records are crucial for developing personalized, systematic, and evidence-based post-surgical management strategies, ultimately improving patient outcomes and quality of life. 5 The prevalent use of free-text for documenting mechanical thrombectomy introduces inefficiencies in medical data utilization. Extracting precise data from such texts is error-prone and incomplete, complicating the integration with structured systems like Electronic Health Records. 6,7 This disjunction not only stymies extensive clinical research but also hinders clinical decision-making and effective patient follow-up, as the information is poorly suited for quick reference or integration into clinical decision support systems. Adopting structured reports for mechanical thrombectomy operations in AIS patients holds considerable clinical benefits. Structured reports enhance the speed and clarity of presenting surgical data, aiding healthcare providers in making informed and timely treatment decisions. Integration of structured reports into clinical decision support systems bolsters the accuracy and scientific basis of -Abstract Truncated-

Evaluating local open-source large language models for data extraction from unstructured reports on mechanical thrombectomy in patients with ischemic stroke

Large language models can effectively extract stroke and reperfusion audit data from medical free-text discharge summaries

Precision Structuring of Free-Text Surgical Record for Enhanced Stroke Management: A Comparative Evaluation of Large Language Models

Automating Stroke Data Extraction From Free-Text Radiology Reports Using Natural Language Processing: Instrument Validation Study

Automated Extraction of Stroke Severity From Unstructured Electronic Health Records Using Natural Language Processing

Data Extraction from Free-Text Reports on Mechanical Thrombectomy in Acute Ischemic Stroke Using ChatGPT: A Retrospective Analysis

Performance of an Open-Source Large Language Model in Extracting Information from Free-Text Radiology Reports

Prediction of large vessel occlusion for ischaemic stroke by using the machine learning model random forests

Natural Language Processing of Radiology Reports to Detect Complications of Ischemic Stroke

From Text to Tables: A Local Privacy Preserving Large Language Model for Structured Information Retrieval from Medical Documents

Development of a Natural Language Processing (NLP) model to automatically extract clinical data from electronic health records: results from an Italian comprehensive stroke center

Interpretable Machine Learning Modeling for Ischemic Stroke Outcome Prediction

Abstract MP32: Machine Learning Models Identify Predictors of Poor Outcome in Patients Undergoing Mechanical Thrombectomy for Acute Ischemic Stroke

A Comparative Analysis of Privacy-Preserving Large Language Models For Automated Echocardiography Report Analysis

Automated Transformation of Unstructured Cardiovascular Diagnostic Reports into Structured Datasets Using Sequentially Deployed Large Language Models

Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports

Privacy-preserving large language models for structured medical information retrieval

Local large language models for privacy-preserving accelerated review of historic echocardiogram reports

Predictors of In-Hospital Mortality after Thrombectomy in Anterior Circulation Large Vessel Occlusion: A Retrospective, Machine Learning Study

Abstract TMP57: User Engagement is Associated With Clinical Impacts of Automated LVO Detection Software

Automating Ischemic Stroke Subtype Classification Using Machine Learning and Natural Language Processing