Evaluation of a Data Annotation Platform for Large, Time-Series Datasets in Intensive Care: Mixed Methods Study

Marceli Wac,Raul Santos-Rodriguez,Chris McWilliams,Christopher Bourdeaux

2024-10-22

Abstract:Intensive Care Units are complex, data-rich environments where critically ill patients are treated using variety of clinical equipment. The data collected using this equipment can be used clinical staff to gain insight into the condition of the patients and provide adequate treatment, but it also provides ample opportunity for applications in machine learning and data science. While this data can frequently be used directly, complex problems may require additional annotations to provide context and meaning before it could be used to train the machine learning models. Annotating time-series datasets in clinical setting is a complex problem due to a large volume and complexity of the data, time-consuming nature of the process and the fact that clinicians' time is in both high demand and short supply. In this study, we present an evaluation of a bespoke tool designed to annotate large, clinical time-series datasets with staff from intensive care units. The software incorporates two modes for annotation: by annotating individual admissions and by generating rulesets which are applied to the entire dataset. Our study was split into two stages focusing on individual and semi-automated annotation and included 28 annotators across both stages who utilised 50 clinical parameters to guide their annotations. We experienced significant challenges in recruitment and engagement of the participants in the annotation activities and developed interventions which improved the participation over the course of the study. During the semi-automated annotation, we observed preferences for different parameter types (measured vs. observed), as well as relative agreement of participants across shared admissions to the decision-tree model trained using their rulesets.

Human-Computer Interaction

What problem does this paper attempt to address?

The problems that this paper attempts to solve are: how to effectively manage and utilize a large amount of complex time - series data in the intensive care unit (ICU), especially the challenges encountered during the data - labeling process. Specifically, the paper focuses on developing and evaluating a data - labeling platform specifically for large - scale time - series datasets to improve the efficiency and quality of data - labeling.

### Specific Background and Challenges of the Problem

1. **Large and Complex Data Volume**:
The amount of data collected in the ICU environment is very large. Each patient may generate more than 1,000 different types of data points. These data include vital signs, drug use, laboratory test results, etc. This large - scale data makes management and effective utilization difficult.

2. **Time - consuming and Labor - intensive Labeling Tasks**:
Data - labeling usually requires clinicians to make judgments based on multiple parameters, medical histories, and laboratory results of patients. This is a highly professional and time - consuming task. Since the time of clinicians is very precious, it is crucial to ensure the effectiveness of the data - labeling workflow.

3. **Limitations of Traditional Labeling Methods**:
Traditional data - labeling methods (such as crowdsourcing) are not applicable in the medical field because personnel with professional knowledge are required to label data, and the time and resources of these people are limited. In addition, traditional methods assume that a large number of labelers label a small part of the data, which is not feasible in medical data - labeling.

4. **The Need for Semi - supervised Learning and Weakly - supervised Methods**:
In order to deal with the difficulty of fully labeling all data, the research proposes the use of semi - supervised learning and weakly - supervised methods. These methods can utilize unlabeled data or automatically generate labels by defining rules (such as label functions), thereby reducing the workload of manual - labeling.

### Research Objectives

The main objective of this paper is to design and evaluate a data - labeling platform for large - scale time - series datasets in the ICU. This platform can:
- Support direct - labeling of individual cases.
- Allow labelers to define rule sets for automatically - labeling the entire dataset.
- Support multi - person collaborative - labeling and be able to handle label conflicts among different labelers.
- Improve the efficiency and accuracy of data - labeling, especially for complex clinical data.

Through these functions, this platform aims to solve the key challenges in ICU data - labeling, provide high - quality labeled data for subsequent machine - learning model training, and thus help improve clinical decision - support systems and other medical applications.