Experimental Evaluation and Development of a Silver-Standard for the MIMIC-III Clinical Coding Dataset

Thomas Searle,Zina Ibrahim,Richard JB Dobson

DOI: https://doi.org/10.18653/v1/2020.bionlp-1.8

2020-06-13

Abstract:Clinical coding is currently a labour-intensive, error-prone, but critical administrative process whereby hospital patient episodes are manually assigned codes by qualified staff from large, standardised taxonomic hierarchies of codes. Automating clinical coding has a long history in NLP research and has recently seen novel developments setting new state of the art results. A popular dataset used in this task is MIMIC-III, a large intensive care database that includes clinical free text notes and associated codes. We argue for the reconsideration of the validity MIMIC-III's assigned codes that are often treated as gold-standard, especially when MIMIC-III has not undergone secondary validation. This work presents an open-source, reproducible experimental methodology for assessing the validity of codes derived from EHR discharge summaries. We exemplify the methodology with MIMIC-III discharge summaries and show the most frequently assigned codes in MIMIC-III are under-coded up to 35%.

Machine Learning

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to evaluate the validity of the MIMIC - III clinical coding dataset, especially the accuracy of its coding. The author points out that although the MIMIC - III dataset is widely used in automatic clinical coding tasks and is often regarded as the "gold standard", in fact, this dataset has not been secondarily verified and may have significant under - coding problems. Specifically, the paper evaluates the validity of the codes extracted from the electronic health record (EHR) discharge summaries by developing an open - source and reproducible experimental method, and taking the MIMIC - III dataset as an example, shows that the most common codes in MIMIC - III may have up to 35% under - coding situations.

### Core problems of the paper
1. **Validity of MIMIC - III coding**:
- Are the codes in the MIMIC - III dataset reliable?
- What is the degree of under - coding?

2. **Challenges of automated clinical coding**:
- Manual clinical coding is a labor - intensive and error - prone process.
- Automated clinical coding systems require high - quality training data, but the quality of existing datasets varies.

3. **Development of experimental methods**:
- How to develop a reproducible experimental method to evaluate the validity of clinical coding?
- How to use the MIMIC - III dataset to generate a silver - standard coding dataset?

### Main findings
- **Under - coding problem**: The study found that there is significant under - coding in the most common codes in the MIMIC - III dataset, up to 35% at most.
- **Effectiveness of experimental methods**: Through text extraction rules and pre - trained named entity recognition and linking tools (such as MedCAT), the accuracy of clinical coding can be effectively evaluated and improved.
- **Generation of datasets**: The study generated a silver - standard clinical coding dataset, providing a reliable benchmark for future automated coding research.

### Conclusion
The paper emphasizes the need to re - evaluate the validity of its coding when using the MIMIC - III dataset for automated clinical coding research. By developing and applying open - source experimental methods, the accuracy and reliability of clinical coding can be improved, thus promoting the development of automated clinical coding technology.