Abstract:Background: The integrity of clinical research and machine learning models in healthcare heavily relies on the quality of underlying clinical laboratory data. However, the preprocessing of this data to ensure its reliability and accuracy remains a significant challenge due to variations in data recording and reporting standards. Methods: We developed lab2clean, a novel algorithm aimed at automating and standardizing the cleaning of retrospective clinical laboratory results data. lab2clean was implemented as two R functions specifically designed to enhance data conformance and plausibility by standardizing result formats and validating result values. The functionality and performance of the algorithm were evaluated using two extensive electronic medical record (EMR) databases, encompassing various clinical settings. Results: lab2clean effectively reduced the variability of laboratory results and identified potentially erroneous records. Upon deployment, it demonstrated effective and fast standardization and validation of substantial laboratory data records. The evaluation highlighted significant improvements in the conformance and plausibility of lab results, confirming the algorithm's efficacy in handling large-scale data sets. Conclusions: lab2clean addresses the challenge of preprocessing and cleaning clinical laboratory data, a critical step in ensuring high-quality data for research outcomes. It offers a straightforward, efficient tool for researchers, improving the quality of clinical laboratory data, a major portion of healthcare data. Thereby, enhancing the reliability and reproducibility of clinical research outcomes and clinical machine learning models. Future developments aim to broaden its functionality and accessibility, solidifying its vital role in healthcare data management.

Automated Data Cleaning For Data Centers: A Case Study

Robust and Automatic Data Cleansing Method for Short-Term Load Forecasting of Distribution Feeders

A Data Fusion and Data Cleaning System for Smart Grids Big Data.

Data Cleaning In Data Warehouse: A Survey of Data Pre-processing Techniques and Tools

A Survey on Data Cleaning Methods for Improved Machine Learning Model Performance

Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach

Data Cleaning for Power Quality Monitoring

Human-Centric Data Cleaning [Vision]

Research on Data Preprocessing Methods for Big Data

Distance-based Data Cleaning: A Survey (Technical Report)

A Data Cleaning Method for Industrial Data Flow Based on Multistage Combinational Optimization of Rule Set

Data Preprocessing: The Techniques for Preparing Clean and Quality Data for Data Analytics Process

Cleanits

A Rule-Based Data Preprocessing Framework for Chiller Rooms Inspired by the Analysis of Engineering Big Data

IHCS: An Integrated Hybrid Cleaning System.

Data Cleaning About Student Information Based on Massive Open Online Course System.

Machine Learning and Data Cleaning: Which Serves the Other?

lab2clean: a novel algorithm for automated cleaning of retrospective clinical laboratory results data for secondary uses

CleanML: A Study for Evaluating the Impact of Data Cleaning on ML Classification Tasks

HITCleaner: A Light-Weight Online Data Cleaning System.

Akane: Perplexity-Guided Time Series Data Cleaning