Abstract:Background: Stroke is a prevalent disease with a significant global impact. Effective assessment of stroke severity is vital for an accurate diagnosis, appropriate treatment, and optimal clinical outcomes. The National Institutes of Health Stroke Scale (NIHSS) is a widely used scale for quantitatively assessing stroke severity. However, the current manual scoring of NIHSS is labor-intensive, time-consuming, and sometimes unreliable. Applying artificial intelligence (AI) techniques to automate the quantitative assessment of stroke on vast amounts of electronic health records (EHRs) has attracted much interest. Objective: This study aims to develop an automatic, quantitative stroke severity assessment framework through automating the entire NIHSS scoring process on Chinese clinical EHRs. Methods: Our approach consists of two major parts: Chinese clinical named entity recognition (CNER) with a domain-adaptive pre-trained large language model (LLM) and automated NIHSS scoring. To build a high-performing CNER model, we first construct a stroke-specific, densely annotated dataset "Chinese Stroke Clinical Records" (CSCR) from EHRs provided by our partner hospital, based on a stroke ontology that defines semantically related entities for stroke assessment. We then pre-train a Chinese clinical LLM coined "CliRoberta" through domain-adaptive transfer learning and construct a deep learning-based CNER model that can accurately extract entities directly from Chinese EHRs. Finally, an automated, end-to-end NIHSS scoring pipeline is proposed by mapping the extracted entities to relevant NIHSS items and values, to quantitatively assess the stroke severity. Results: Results obtained on a benchmark dataset CCKS2019 and our newly created CSCR dataset demonstrate the superior performance of our domain-adaptive pre-trained LLM and the CNER model, compared with the existing benchmark LLMs and CNER models. The high F1 score of 0.990 ensures the reliability of our model in accurately extracting the entities for the subsequent automatic NIHSS scoring. Subsequently, our automated, end-to-end NIHSS scoring approach achieved excellent inter-rater agreement (0.823) and intraclass consistency (0.986) with the ground truth and significantly reduced the processing time from minutes to a few seconds. Conclusion: Our proposed automatic and quantitative framework for assessing stroke severity demonstrates exceptional performance and reliability through directly scoring the NIHSS from diagnostic notes in Chinese clinical EHRs. Moreover, this study also contributes a new clinical dataset, a pre-trained clinical LLM, and an effective deep learning-based CNER model. The deployment of these advanced algorithms can improve the accuracy and efficiency of clinical assessment, and help improve the quality, affordability and productivity of healthcare services.

The Development of Personalized Writing Assistant for Electronic Discharge Summaries Based on Named Entity Recognition

A Study on Automatic Generation of Chinese Discharge Summary.

A Template-Driven Framework for Chinese Medical Named Entity Recognition

NLP to Improve Accuracy and Quality of Dictated Medical Documents-Final Report

A comprehensive study of named entity recognition in Chinese clinical text

AI Managed Emergency Documentation with a Pretrained Model

Chinese Clinical Named Entity Recognition with Word-Level Information Incorporating Dictionaries

Constructing a Chinese Electronic Medical Record Corpus for Named Entity Recognition on Resident Admit Notes

Deep learning for named entity recognition on Chinese electronic medical records: Combining deep transfer learning with multitask bi-directional LSTM RNN

Effidit: Your AI Writing Assistant

A query interface for clinical research with Chinese electronic health record using Natural Language Processing

A multitask bi-directional RNN model for named entity recognition on Chinese electronic medical records

A Deep Learning Model Incorporating Part of Speech and Self-Matching Attention for Named Entity Recognition of Chinese Electronic Medical Records

Effidit: an Assistant for Improving Writing Efficiency.

Write-righter: an Academic Writing Assistant System

Named Entity Recognition of Chinese Electronic Medical Records Based on Multifeature Embedding and Attention Mechanism

Named Entity Recognition in Chinese Electronic Medical Records Based on CRF.

Named Entity Recognition from Chinese Adverse Drug Event Reports with Lexical Feature Based BiLSTM-CRF and Tri-Training

Named Entity Recognition In Chinese Electronic Medical Record Using Attention Mechanism

Chinese medical entity recognition based on the dual-branch TENER model

Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model