Abstract:Quantitative data cleaning relies on the use of statistical methods to identify and repair data quality problems while logical data cleaning tackles the same problems using various forms of logical reasoning over declarative dependencies. Each of these approaches has its strengths: the logical approach is able to capture subtle data quality problems using sophisticated dependencies, while the quantitative approach excels at ensuring that the repaired data has desired statistical properties. We propose a novel framework within which these two approaches can be used synergistically to combine their respective strengths. We instantiate our framework using (i) metric functional dependencies, a type of dependency that generalizes functional dependencies (FDs) to identify inconsistencies in domains where only large differences in metric data are considered to be a data quality problem, and (ii) repairs that modify the inconsistent data so as to minimize statistical distortion, measured using the Earth Mover's Distance. We show that the problem of computing a statistical distortion minimal repair is NP-hard. Given this complexity, we present an efficient algorithm for finding a minimal repair that has a small statistical distortion using EMD computation over semantically related attributes. To identify semantically related attributes, we present a sound and complete axiomatization and an efficient algorithm for testing implication of metric FDs. While the complexity of inference for some other FD extensions is co-NP complete, we show that the inference problem for metric FDs remains linear, as in traditional FDs. We prove that every instance that can be generated by our repair algorithm is set-minimal (with no unnecessary changes). Our experimental evaluation demonstrates that our techniques obtain a considerably lower statistical distortion than existing repair techniques, while achieving similar levels of efficiency.

Using Fuzzy Logic to Evaluate Normalization Completeness for An Improved Database Design

Normalization Based On Fuzzy Functional Dependency In A Fuzzy Relational Data Model

Assessing Data Quality Within Available Context

An Extended Boyce-Codd Normal Form in Fuzzy Relational Databases

Quantification for the Importance Degree of Engineering Characteristics with a Multi-Level Hierarchical Structure in QFD

Embedded functional dependencies and data-completeness tailored database design

Fuzzy Functional Dependency and a Series of Design Issues of Fuzzy Relational Databases

Determining the Real Data Completeness of a Relational Dataset

Conceptual Design of Fuzzy Object-Oriented Databases Using Extended Entity-Relationship Model.

Fuzzy Logic in Data Modeling: Semantics, Constraints, and Database Design

Dominance-based Fuzzy Rough Set Approach for Incomplete Interval-Valued Data

Correctness Criteria for Normalization of Semistructured Data.

A Computational Algorithm for the Ffd Transitive Closure and A Complete Axiomatization of Fuzzy Functional Dependence (Ffd)

A Logical Design Methodology for Relational Databases Using the Extended Entity-Relationship Model

Measurable & Scalable NFRs using Fuzzy Logic and Likert Scale

Prioritizing Technical Debt in Database Normalization Using Portfolio Theory and Data Quality Metrics

Paraconsistent Intuitionistic Fuzzy Relational Data Model

Identifying and Managing Technical Debt in Database Normalization Using Machine Learning and Trade-off Analysis

On the Lossless-Join Decomposition of Relation Scheme(s) in a Fuzzy Relational Data Model

On Desirable Semantics of Functional Dependencies over Databases with Incomplete Information

Combining quantitative and logical data cleaning