Abstract:With the rapid development of the internet technology, dirty data are commonly observed in various real scenarios, e.g., owing to unreliable sensor reading, transmission and collection from heterogeneous sources. To deal with their negative effects on downstream applications, data cleaning approaches are designed to preprocess the dirty data before conducting applications. The idea of most data cleaning methods is to identify or correct dirty data, referring to the values of their neighbors which share the same information. Unfortunately, owing to data sparsity and heterogeneity, the number of neighbors based on equality relationship is rather limited, especially in the presence of data values with variances. To tackle this problem, distance-based data cleaning approaches propose to consider similarity neighbors based on value distance. By tolerance of small variants, the enriched similarity neighbors can be identified and used for data cleaning tasks. At the same time, distance relationship between tuples is also helpful to guide the data cleaning, which contains more information and includes the equality relationship. Therefore, distance-based technology plays an important role in the data cleaning area, and we also have reason to believe that distance-based data cleaning technology will attract more attention in data preprocessing research in the future. Hence this survey provides a classification of four main data cleaning tasks, i.e., rule profiling, error detection, data repair and data imputation, and comprehensively reviews the state of the art for each class.

Data Cleaning In Data Warehouse: A Survey of Data Pre-processing Techniques and Tools

Automated Data Cleaning For Data Centers: A Case Study

Distance-based Data Cleaning: A Survey (Technical Report)

Data Preprocessing: The Techniques for Preparing Clean and Quality Data for Data Analytics Process

A Survey on Data Cleaning Methods for Improved Machine Learning Model Performance

Reduce The Wastage of Data During Movement in Data Warehouse

Data Cleaning for Power Quality Monitoring

Research on Data Preprocessing Methods for Big Data

A Peta-Scale Data Movement and Analysis in Data Warehouse (APSDMADW)

Data Quality Measures and Data Cleansing for Research Information Systems

Extraction, Transformation, and Loading in a Data Warehouse Course

Drilling data quality improvement and information extraction with case studies

The technology of using a data warehouse to support decision-making in health care

A Primer on the Data Cleaning Pipeline

A Simplified Approach for Quality Management in Data Warehouse

Data Quality Assessment: Challenges and Opportunities

A Data Warehouse Design for a Typical University Information System

Data warehouse on Manpower Employment for Decision Support System

Security Issues in Data Warehouse

Modeling ETL Process for Data Warehouse: An Exploratory Study

Data Context Informed Data Wrangling