Abstract:Most theoretical frameworks that focus on data errors and inconsistencies follow logic-based reasoning. Yet, practical data cleaning tools need to incorporate statistical reasoning to be effective in real-world data cleaning tasks. Motivated by these empirical successes, we propose a formal framework for unclean databases, where two types of statistical knowledge are incorporated: The first represents a belief of how intended (clean) data is generated, and the second represents a belief of how noise is introduced in the actual observed database instance. To capture this noisy channel model, we introduce the concept of a Probabilistic Unclean Database (PUD), a triple that consists of a probabilistic database that we call the intention, a probabilistic data transformator that we call the realization and captures how noise is introduced, and a dirty observed database instance that we call the observation. We define three computational problems in the PUD framework: cleaning (infer the most probable clean instance given a PUD), probabilistic query answering (compute the probability of an answer tuple over the unclean observed instance), and learning (estimate the most likely intention and realization models of a PUD given a collection of training data). We illustrate the PUD framework on concrete representations of the intention and realization, show that they generalize traditional concepts of repairs such as cardinality and value repairs, draw connection to consistent query answering, and prove tractability results. We further show that parameters can be learned in practical instantiations, and in fact, prove that under certain conditions we can learn a PUD directly from a single dirty database instance without any need for clean examples.

Bayesian Data Cleaning for Web Data

Data Cleaning Using Belief Propagation

A data cleaning framework based on user feedback

Distance-based Data Cleaning: A Survey (Technical Report)

A probabilistic database approach to autoencoder-based data cleaning

A Formal Framework For Probabilistic Unclean Databases

Batchwise Probabilistic Incremental Data Cleaning

ActiveClean: Interactive Data Cleaning While Learning Convex Loss Models

Exploring Artificial Intelligence Architecture in Data Cleaning Based on Bayesian Networks

An Ontology-Based Approach to Data Cleaning

Crowdcleaner: A Data Cleaning System Based on Crowdsourcing

Data repair of density-based data cleaning approach using conditional functional dependencies

CleanML: A Study for Evaluating the Impact of Data Cleaning on ML Classification Tasks

Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach

A Hybrid Data Cleaning Framework Using Markov Logic Networks

RetClean: Retrieval-Based Data Cleaning Using Foundation Models and Data Lakes

A Survey on Data Cleaning Methods for Improved Machine Learning Model Performance

Human-Centric Data Cleaning [Vision]

Robust Bayesian Networks for Low-Quality Data Modeling and Process Monitoring Applications

Akane: Perplexity-Guided Time Series Data Cleaning

Guided conditional functional dependency discovery