Abstract:With the rapid development of the internet technology, dirty data are commonly observed in various real scenarios, e.g., owing to unreliable sensor reading, transmission and collection from heterogeneous sources. To deal with their negative effects on downstream applications, data cleaning approaches are designed to preprocess the dirty data before conducting applications. The idea of most data cleaning methods is to identify or correct dirty data, referring to the values of their neighbors which share the same information. Unfortunately, owing to data sparsity and heterogeneity, the number of neighbors based on equality relationship is rather limited, especially in the presence of data values with variances. To tackle this problem, distance-based data cleaning approaches propose to consider similarity neighbors based on value distance. By tolerance of small variants, the enriched similarity neighbors can be identified and used for data cleaning tasks. At the same time, distance relationship between tuples is also helpful to guide the data cleaning, which contains more information and includes the equality relationship. Therefore, distance-based technology plays an important role in the data cleaning area, and we also have reason to believe that distance-based data cleaning technology will attract more attention in data preprocessing research in the future. Hence this survey provides a classification of four main data cleaning tasks, i.e., rule profiling, error detection, data repair and data imputation, and comprehensively reviews the state of the art for each class.

Exploring Artificial Intelligence Architecture in Data Cleaning Based on Bayesian Networks

The Data Cleaning of Electric Industrial Control Terminal Based on the Iforest and Genetic BP Neural Network Algorithms

Using Visualization to Improve Clustering Analysis on Heterogeneous Information Network.

Bayesian Data Cleaning for Web Data

Distance-based Data Cleaning: A Survey (Technical Report)

Akane: Perplexity-Guided Time Series Data Cleaning

A Novel Data Cleaning Framework Based on Knowledge Graph

Toward a view-based data cleaning architecture

A Healthy Big Data Classification Model Based on Improved Bayesian Network

The Optimization of the Big Data Cleaning Based on Task Merging

A Data Cleaning Method for Industrial Data Flow Based on Multistage Combinational Optimization of Rule Set

Research and Citation Analysis of Data Mining Technology Based on Bayes Algorithm

Semantic-based Intelligent Data Clean Framework for Big Data

Application of AI in Computer Network Technology in the Big Data Era

Data Cleaning Using Belief Propagation

Learning the structure of Bayesian networks with ancestral and/or heuristic partition

A Hybrid Data Cleaning Framework Using Markov Logic Networks

Impacts of Dirty Data: and Experimental Evaluation

CleanML: A Study for Evaluating the Impact of Data Cleaning on ML Classification Tasks

Research on an advanced intelligence implementation system for engineering process in industrial field under big data

Batchwise Probabilistic Incremental Data Cleaning