Abstract:With the rapid development of the internet technology, dirty data are commonly observed in various real scenarios, e.g., owing to unreliable sensor reading, transmission and collection from heterogeneous sources. To deal with their negative effects on downstream applications, data cleaning approaches are designed to preprocess the dirty data before conducting applications. The idea of most data cleaning methods is to identify or correct dirty data, referring to the values of their neighbors which share the same information. Unfortunately, owing to data sparsity and heterogeneity, the number of neighbors based on equality relationship is rather limited, especially in the presence of data values with variances. To tackle this problem, distance-based data cleaning approaches propose to consider similarity neighbors based on value distance. By tolerance of small variants, the enriched similarity neighbors can be identified and used for data cleaning tasks. At the same time, distance relationship between tuples is also helpful to guide the data cleaning, which contains more information and includes the equality relationship. Therefore, distance-based technology plays an important role in the data cleaning area, and we also have reason to believe that distance-based data cleaning technology will attract more attention in data preprocessing research in the future. Hence this survey provides a classification of four main data cleaning tasks, i.e., rule profiling, error detection, data repair and data imputation, and comprehensively reviews the state of the art for each class.

Research Status and Prospect of Data Extraction and Cleaning Technology in Large Environment

Research on Data Preprocessing Methods for Big Data

Research of Agricultural Big data

Research and Prospect of Big Data Audit Technology

Research on the application of big data technology in enterprise project management

Research situation and expectation of clinical healthcare big data

Weeding Method of Traditional Chinese Medicine Planting Based on Big Data Analysis

Research on big data processing of water conservancy automation

A Data Cleaning Method for Industrial Data Flow Based on Multistage Combinational Optimization of Rule Set

Application and Prospect of Big Data Technology in Medical Image Enhancement

Applied Research on Agricultural Big Data

Research on Data Cleaning Algorithm Based on Multi Type Construction Waste

Distance-based Data Cleaning: A Survey (Technical Report)

Current Situation and Development Direction of Medical Image Processing Technology in the Background of Big Data

Survey of Data Annotation

Research on Key Information Processing Technology of Environmental Law Enforcement Based on Big Data

Research on Urban Planning Method from Big Data Perspective

Big Data In Construction Waste Management: Prospects And Challenges

Impacts on environmental quality and required environmental regulation adjustments: A perspective of directed technical change driven by big data

Research on Development of Health Management Industry Based on Big Data Technology

Research on Urban and Rural Planning and Smart City Construction in Big Data Era