Precision Structuring of Free-Text Surgical Record for Enhanced Stroke Management: A Comparative Evaluation of Large Language Models
Mengfei Wang,Jianyong Wei,Yao Zeng,Lisong Dai,Bicong Yan,Yueqi Zhu,Xiaoer Wei,Yidong Jin,Yuehua Li
DOI: https://doi.org/10.2147/jmdh.s486449
2024-11-17
Journal of Multidisciplinary Healthcare
Abstract:Mengfei Wang, 1 Jianyong Wei, 2 Yao Zeng, 3 Lisong Dai, 4 Bicong Yan, 4 Yueqi Zhu, 4 Xiaoer Wei, 4 Yidong Jin, 1 Yuehua Li 4 1 School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, People's Republic of China; 2 Clinical Research Center, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, People's Republic of China; 3 Shenyang University of Technology, Shenyang, Liaoning, People's Republic of China; 4 Institute of Diagnostic and Interventional Radiology, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, People's Republic of China Correspondence: Yuehua Li, Institute of Diagnostic and Interventional Radiology, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, No. 600 Yishan Road, Shanghai, 200233, People's Republic of China, Email Introduction: Mechanical thrombectomy (MTB) is a critical procedure for acute ischemic stroke (AIS) patients. However, the free-text format of MTB surgical records limits the formulation of effective postoperative patient management and rehabilitation plans. This study compares the efficacy of large language models (LLMs) in structuring data from these free-text MTB surgical record. Methods: This retrospective study collected a total of 382 MTB surgical records from a tertiary hospital. An initial analysis of 30 surgical record from these records provided a guiding prompt for LLMs, focusing on basic and advanced characteristics, such as occlusion locations, thrombectomy maneuvers, reperfusion status, and intraoperative complications. Six LLMs—ChatGPT, GPT-4, GeminiPro, ChatGLM4, Spark3, and QwenMax—were assessed against data extracted by neuroradiologists and a junior physician for comparison. The all 382 surgical records were used to test the performance of LLMs. The performance of the LLMs was quantified using Accuracy, Sensitivity, Specificity, AUC, and MSE as an additional metric for advanced characteristics. Results: All LLMs showed high performance in characteristic extraction, achieving an average accuracy of 95.09 ± 4.98% across 48 items, and 78.05 ± 4.2% overall. GLM4 and GPT-4 were most accurate in advanced characteristics extraction, with accuracies of 84.03% and 82.20%, respectively. The processing time for LLMs averaged 73.10 ± 10.86 seconds of six models, significantly faster than the 427.88 seconds for manual extraction by physicians. Conclusion: LLMs, particularly GLM4 and GPT-4, efficiently and accurately structured both general and advanced characteristics from MTB surgical record, outperforming manual extraction methods and demonstrating potential for enhancing clinical data management in AIS treatment. Keywords: large language models, free-text report, mechanical thrombectomy, acute ischemia stroke Acute Ischemic Stroke (AIS) stands as a leading cause of mortality and debilitating conditions among adults globally. 1 The burden of stroke has increased significantly over recent decades, with the Global Burden of Disease (GBD) study showing that the annual number of strokes and stroke-related deaths increased substantially from 1990 to 2019. 2 This trend is particularly evident in China, where the burden of stroke remains critically high. 3,4 The surgical records of mechanical thrombectomy (MTB) detail essential intraoperative findings, including the extent of blood flow restoration and reperfusion status. These records are crucial for developing personalized, systematic, and evidence-based post-surgical management strategies, ultimately improving patient outcomes and quality of life. 5 The prevalent use of free-text for documenting mechanical thrombectomy introduces inefficiencies in medical data utilization. Extracting precise data from such texts is error-prone and incomplete, complicating the integration with structured systems like Electronic Health Records. 6,7 This disjunction not only stymies extensive clinical research but also hinders clinical decision-making and effective patient follow-up, as the information is poorly suited for quick reference or integration into clinical decision support systems. Adopting structured reports for mechanical thrombectomy operations in AIS patients holds considerable clinical benefits. Structured reports enhance the speed and clarity of presenting surgical data, aiding healthcare providers in making informed and timely treatment decisions. Integration of structured reports into clinical decision support systems bolsters the accuracy and scientific basis of -Abstract Truncated-
health care sciences & services