Integrity Constraints on Rich Data Types

Shaoxu Song,Lei Chen

DOI: https://doi.org/10.1007/978-3-031-27177-9

2023-01-01

Synthesis Lectures on Data Management

Abstract:This book provides a comprehensive guide to the application of data dependencies to myriad big data analysis issues

What problem does this paper attempt to address?

Data Dependencies over Big Data: A Family Tree

Shaoxu Song,Fei Gao,Ruihong Huang,Chaokun Wang

DOI: https://doi.org/10.1109/tkde.2020.3046443

IF: 9.235

2020-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Besides the conventional schema-oriented tasks, data dependencies are recently revisited for data quality applications, such as violation detection, data repairing and record matching. To address the variety and veracity issues of big data, data dependencies have been extended as data quality rules to adapt to various data types, ranging from (1) categorical data with equality relationships to (2) heterogeneous data with similarity relationships, and (3) numerical data with order relationships. In this survey, we briefly review the recent proposals on data dependencies categorized into the aforesaid types of data. In addition to (a) the concepts of these data dependency notations, we investigate (b) the extension relationships between data dependencies, e.g., conditional functional dependencies (CFDs) extend the conventional functional dependencies (FDs). It forms a family tree of extensions, mostly rooted in FDs, helping us understand the expressive power of various data dependencies. Moreover, we summarize (c) the discovery of dependencies from data, since data dependencies are often unlikely to be manually specified in a traditional way, given the huge volume and high variety of big data. We further outline (d) the applications of the extended data dependencies, in particular in data quality practice. It guides users to select proper data dependencies with sufficient expressive power and reasonable discovery cost. Finally, we conclude with several directions of future studies on the emerging data.
Data Dependencies Extended for Variety and Veracity: A Family Tree

Shaoxu Song,Fei Gao,Ruihong Huang,Chaokun Wang

DOI: https://doi.org/10.1109/tkde.2020.3046443

2023-01-01

Abstract:Besides the conventional schema-oriented tasks, data dependencies are recently revisited for data quality applications, such as violation detection, data repairing and record matching. To address the variety and veracity issues of big data, data dependencies have been extended as data quality rules to adapt to various data types, ranging from (1) categorical data with equality relationships to (2) heterogeneous data with similarity relationships, and (3) numerical data with order relationships. In this survey, we briefly review the recent proposals on data dependencies categorized into the aforesaid types of data. In addition to (a) the concepts of these data dependency notations, we investigate (b) the extension relationships between data dependencies, e.g., conditional functional dependencies (CFDs) extend the conventional functional dependencies (FDs). It forms a family tree of extensions, mostly rooted in FDs, helping us understand the expressive power of various data dependencies. Moreover, we summarize (c) the discovery of dependencies from data, since data dependencies are often unlikely to be manually specified in a traditional way, given the huge volume and high variety of big data. We further outline (d) the applications of the extended data dependencies, in particular in data quality practice. It guides users to select proper data dependencies with sufficient expressive power and reasonable discovery cost. Finally, we conclude with several directions of future studies on the emerging data.
Data Dependencies Extended for Variety and Veracity: A Family Tree (extended Abstract)

Shaoxu Song,Fei Gao,Ruihong Huang,Chaokun Wang

DOI: https://doi.org/10.1109/icde55515.2023.00336

2023-01-01

Abstract:To address the variety and veracity issues of big data, data dependencies have been extended as data quality rules to adapt to various data types, ranging from (1) categorical data with equality relationships to (2) heterogeneous data with similarity relationships, and (3) numerical data with order relationships. In this survey, we briefly review the recent proposals on data dependencies categorized into the aforesaid types of data. In addition to (a) the concepts of these data dependency notations, we investigate (b) the extension relationships between data dependencies. It forms a family tree of extensions, mostly rooted in FDs. Moreover, we summarize (c) the discovery of dependencies from data, and (d) the applications of the extended data dependencies. Finally, we conclude with several directions of future studies on the emerging data.
Using Semantic Web Technologies to Specify Constraints of RBAC

Wu Di,Jian Lin,Yabo Dong,Miaoliang Zhu

DOI: https://doi.org/10.1109/PDCAT.2005.247

2005-01-01

Abstract:Role-based access control (RBAC) models have generated a great interest in the security community as a powerful and generalized approach to security management. One of important aspects in RBAC is constraints that constrain what components in RBAC are allowed to do. There are lots of research have been achieved to specify constraints for secure system developers. However more work is need urgently to met requirements for interoperability of machine and people understandable constraints specification in open and distributed environment. In this paper we propose another approach to specify constraints using Semantic Web technologies. The Web Ontology Language (OWL) specification of basic RBAC components and constraints are described in detail.
Entity integrity management under data volume, variety and veracity

Sebastian Link

DOI: https://doi.org/10.1007/s10115-022-01814-1

IF: 2.7

2023-01-27

Knowledge and Information Systems

Abstract:Edgar Codd introduced the principle of entity integrity in the context of his relational model of data. The principle says that every targeted real-world entity should be uniquely represented in the database. In actual database systems, entity integrity is typically enforced by primary keys. We introduce a framework toward generalizing entity integrity to different dimensions of data, including volume, variety, and veracity. We establish axiomatic and algorithmic foundations for the implication problem of the combined class of uniqueness constraints, functional dependencies, and multivalued dependencies in all combinations of the dimensions we consider. These are based on specific approaches to the semantics of these integrity constraints and to the dimensions of data. We also highlight how our concepts lead to new opportunities for diverse and important areas of applications, such as query optimization, database design and security, and data quality. Overall, this sets out an agenda for future research that extends our approaches or applies different approaches in this area, as driven by application requirements.

computer science, information systems, artificial intelligence
Integrity Constraints for Data Integration and Exchange Based on XML

Liu Yunfeng,Yang Dongqing,Tang Shiwei,Wang Tengjiao,GAO Jun,LI Ying

DOI: https://doi.org/10.3969/j.issn.1000-3428.2005.09.015

2005-01-01

Abstract:This paper proposes three new XML integrity constraint techniques in CoXML system: Translation of functional dependencies in data exchange based on XML; XML schema oriented key constraints translation; validating key constraints over XML document using XPath. The experimental results conducted on CoXML reveal that these techniques work well in practice.
Incorporating Integrity Constraints in Uncertain Databases

Naveen Ashish,Sharad Mehrotra,Pouria Pirzadeh

DOI: https://doi.org/10.48550/arXiv.0907.1632

2009-07-09

Databases

Abstract:We develop an approach to incorporate additional knowledge, in the form of general purpose integrity constraints (ICs), to reduce uncertainty in probabilistic databases. While incorporating ICs improves data quality (and hence quality of answers to a query), it significantly complicates query processing. To overcome the additional complexity, we develop an approach to map an uncertain relation U with ICs to another uncertain relation U', that approximates the set of consistent worlds represented by U. Queries over U can instead be evaluated over U' achieving higher quality (due to reduced uncertainty in U') without additional complexity in query processing due to ICs. We demonstrate the effectiveness and scalability of our approach to large data-sets with complex constraints. We also present experimental results demonstrating the utility of incorporating integrity constraints in uncertain relations, in the context of an information extraction application.
AUDITOR: A System Designed for Automatic Discovery of Complex Integrity Constraints in Relational Databases.

Wentao Hu,Dongxiang Zhang,Dawei Jiang,Sai Wu,Ke Chen,Kian-Lee Tan,Gang Chen

DOI: https://doi.org/10.1145/3318464.3384683

2020-01-01

Abstract:In this demonstration, we present a new definition of integrity constraint that is more powerful for anomalous data discovery. In our definition, a constraint is functioned on both categorical and numerical attributes in relational tables, as well as their derivative attributes, leading to a huge search space. Furthermore, we are the first to take into account attribute value distribution as part of a constraint. Based on the proposed integrity constraint, we build AUDITOR on top of relational tables from the industry of healthcare auditing and demonstrate its effectiveness and ease-of-use for domain experts to discover anomalous data.
On Data Dependencies in Dataspaces

Shaoxu Song,Lei Chen,Philip S. Yu

DOI: https://doi.org/10.1109/icde.2011.5767857

2011-01-01

Abstract:To study data dependencies over heterogeneous data in dataspaces, we define a general dependency form, namely comparable dependencies (CDs), which specifies constraints on comparable attributes. It covers the semantics of a broad class of dependencies in databases, including functional dependencies (FDs), metric functional dependencies (MFDs), and matching dependencies (MDs). As we illustrated, comparable dependencies are useful in real practice of dataspaces, e.g., semantic query optimization. Due to the heterogeneous data in dataspaces, the first question, known as the validation problem, is to determine whether a dependency (almost) holds in a data instance. Unfortunately, as we proved, the validation problem with certain error or confidence guarantee is generally hard. In fact, the confidence validation problem is also NP-hard to approximate to within any constant factor. Nevertheless, we develop several approaches for efficient approximation computation, including greedy and randomized approaches with an approximation bound on the maximum number of violations that an object may introduce. Finally, through an extensive experimental evaluation on real data, we verify the superiority of our methods.
Constraints, Inference Channels and Secure Databases.

Alexander Brodsky,Csilla Farkas,Duminda Wijesekera,X. Sean Wang

DOI: https://doi.org/10.1007/3-540-45349-0_9

2000-01-01

Abstract:This paper investigates the problem of confidentiality violations via illegal data inferences that occur when arithmetic constraints are combined with non-confidential numeric data to infer confidential information. The database is represented as a point in an ( n + k )-dimensional constraint space, where n is the number of numerical data items stored in the database (extensional database) and k is the number of derivable attributes (intensional database). Database constraints over both extensional and intensional databases form an ( n+ k )-dimensional constraint object. A query answer over a data item x is an interval I of values along the x axis of the database such that I is correct (i.e., the actual data value is within I ) and safe (i.e., users cannot infer which point within I is the actual data value). The security requirements are expressed by the accuracy with which users are allowed to disclose data items. More specifically, we develop two classification methods: (1) volume-based classification, where the entire volume of the disclosed constraint object that contains the data item is considered and (2) interval based classification, where the length of the interval that contains the data item is considered. We develop correct and safe inference algorithms for both cases.
Integrity Constraints for General-Purpose Knowledge Bases

Luís Cruz-Filipe,Isabel Nunes,Peter Schneider-Kamp

DOI: https://doi.org/10.1007/978-3-319-30024-5_13

2016-01-20

Abstract:Integrity constraints in databases have been studied extensively since the 1980s, and they are considered essential to guarantee database integrity. In recent years, several authors have studied how the same notion can be adapted to reasoning frameworks, in such a way that they achieve the purpose of guaranteeing a system's consistency, but are kept separate from the reasoning mechanisms. In this paper we focus on multi-context systems, a general-purpose framework for combining heterogeneous reasoning systems, enhancing them with a notion of integrity constraints that generalizes the corresponding concept in the database world.

Databases
Embedded functional dependencies and data-completeness tailored database design

Ziheng Wei,Sebastian Link

DOI: https://doi.org/10.14778/3342263.3342626

IF: 2.5

2019-07-01

Proceedings of the VLDB Endowment

Abstract:We establish a robust schema design framework for data with missing values. The framework is based on the new notion of an embedded functional dependency, which is independent of the interpretation of missing values, able to express completeness and integrity requirements on application data, and capable of capturing many redundant data value occurrences. We establish axiomatic and algorithmic foundations for reasoning about embedded functional dependencies. These foundations allow us to establish generalizations of Boyce-Codd and Third normal forms that do not permit any redundancy in any future application data, or minimize their redundancy across dependency-preserving decompositions, respectively. We show how to transform any given schema into application schemata that meet given completeness and integrity requirements and the conditions of the generalized normal forms. Data over those application schemata are therefore fit for purpose by design. Extensive experiments with benchmark schemata and data illustrate our framework, and the effectiveness and efficiency of our algorithms, but also provide quantified insight into database schema design trade-offs.

computer science, information systems, theory & methods
Introducing Keys and Integrity Rules in a Fuzzy Relational Data Model

GQ Chen,EE Kerre,J Vandenbulcke

DOI: https://doi.org/10.1109/icsmc.1996.569790

1996-01-01

Abstract:This paper discusses the fuzzy functional dependency (FFD), which is considered as integrity constraints and used to define /spl theta/-keys. Furthermore, weak/strong /spl theta/-entity integrity rules (/spl theta/-EIRs) are defined as a result of the extension of classical entity integrity rule, and /spl theta/-referential integrity rule (/spl theta/-RIR) is defined as a result of the extension of classical referential integrity rule. These results are deemed important, theoretically and practically, both for integrity constraint enforcement and for fuzzy database design.
Discovering Matching Dependencies

Shaoxu Song,Lei Chen

DOI: https://doi.org/10.1145/1645953.1646135

2009-01-01

Abstract:Matching dependencies (MDs) are recently proposed for various data quality applications such as detecting the violation of integrity constraints and duplicate object identification. In this paper, we study the problem of discovering matching dependencies for a given database instance. First, we formally define the measures, support and confidence, for evaluating the utility of MDs in the given database instance. Then, we study the discovery of MDs with certain utility requirements of support and confidence. Exact algorithms are developed, together with pruning strategies to improve the time performance. Finally, our experimental evaluation demonstrates the efficiency of the proposed methods.
Detecting Data Errors with Statistical Constraints

Jing Nathan Yan,Oliver Schulte,Jiannan Wang,Reynold Cheng

DOI: https://doi.org/10.48550/arXiv.1902.09711

2019-02-26

Abstract:A powerful approach to detecting erroneous data is to check which potentially dirty data records are incompatible with a user's domain knowledge. Previous approaches allow the user to specify domain knowledge in the form of logical constraints (e.g., functional dependency and denial constraints). We extend the constraint-based approach by introducing a novel class of statistical constraints (SCs). An SC treats each column as a random variable, and enforces an independence or dependence relationship between two (or a few) random variables. Statistical constraints are expressive, allowing the user to specify a wide range of domain knowledge, beyond traditional integrity constraints. Furthermore, they work harmoniously with downstream statistical modeling. We develop CODED, an SC-Oriented Data Error Detection system that supports three key tasks: (1) Checking whether an SC is violated or not on a given dataset, (2) Identify the top-k records that contribute the most to the violation of an SC, and (3) Checking whether a set of input SCs have conflicts or not. We present effective solutions for each task. Experiments on synthetic and real-world data illustrate how SCs apply to error detection, and provide evidence that CODED performs better than state-of-the-art approaches.

Databases
Approximate Integrity Constraints in Incomplete Databases With Limited Domains

Munqath Al-atar,Attila Sali

2024-02-08

Abstract:In case of incomplete database tables, a possible world is obtained by replacing any missing value by a value from the corresponding attribute's domain that can be infinite. A possible key or possible functional dependency constraint is satisfied by an incomplete table if we can obtain a possible world that satisfies the given key or functional dependency. On the other hand, a certain key or certain functional dependency holds if all possible worlds satisfy the constraint, A strongly possible constraint is an intermediate concept between possible and certain constraints, based on the strongly possible world approach (a strongly possible world is obtained by replacing \nul's by a value from the ones appearing in the corresponding attribute of the table). A strongly possible key or functional dependency holds in an incomplete table if there exists a strongly possible world that satisfies the given constraint. In the present paper, we introduce strongly possible versions of multivalued dependencies and cross joins, and we analyse the complexity of checking the validity of a given strongly possible cross joins. We also study approximation measures of strongly possible keys (spKeys), functional dependencies (spFDs), multivalued dependencies (spMVDs) and cross joins (spCJs). We also treat complexity questions of determination of the approximation values.

Databases
Data dependencies in the presence of difference

Lei Chen,Shaoxu Song

DOI: https://doi.org/10.14711/thesis-b1115869

2010-01-01

Abstract:The importance of difference semantics (e.g., “similar” or “dissimilar”) is recently recognized for declaring dependencies among various types of data, such as numerical values or text values. We propose a novel form of differential dependencies (DDs), which specifies constraints on difference, instead of equality function in traditional dependency notations like functional dependencies. Informally, a differential dependency states that if two tuples have distances on attributes X agreeing with a certain differential function, then their distances on attributes Y should also agree with the corresponding differential function on Y. For example, [date(≤ 7)]→[price(In this thesis, we first report our preliminary work on several theoretical issues of differential dependencies, including formal definitions of DDs and differential keys, subsumption order relation of differential functions, implication of DDs, closure of a differential function, a sound and complete inference system, and minimal cover for DDs. Then, we investigate a practical problem, i.e., how to discover DD s and differential keys from a given sample data. Due to the intrinsic hardness, we develop several pruning methods to improve the discovery efficiency in practice. Moreover, we address differential dependencies that “almost” hold in a given data instance, namely approximate differential dependencies . Several approaches are studied to evaluate how a DD approximately holds in a data instance. Through the extensive experimental evaluation on real data sets, we demonstrate the performance of discovery algorithms, and the effectiveness of DDs in several real applications.
Comparable Dependencies over Heterogeneous Data.

Shaoxu Song,Lei Chen,Philip S. Yu

DOI: https://doi.org/10.1007/s00778-012-0285-7

2012-01-01

The VLDB Journal

Abstract:To study the data dependencies over heterogeneous data in dataspaces, we define a general dependency form, namely comparable dependencies (CDS), which specifies constraints on comparable attributes. It covers the semantics of a broad class of dependencies in databases, including functional dependencies (FDS), metric functional dependencies (MFDS), and matching dependencies (MDS). As we illustrated, comparable dependencies are useful in real practice of dataspaces, such as semantic query optimization. Due to heterogeneous data in dataspaces, the first question, known as the validation problem, is to tell whether a dependency (almost) holds in a data instance. Unfortunately, as we proved, the validation problem with certain error or confidence guarantee is generally hard. In fact, the confidence validation problem is also NP-hard to approximate to within any constant factor. Nevertheless, we develop several approaches for efficient approximation computation, such as greedy and randomized approaches with an approximation bound on the maximum number of violations that an object may introduce. Finally, through an extensive experimental evaluation on real data, we verify the superiority of our methods.
Consistency Maintenance for Constraint in Role-Based Access Control Model

Wei-li Han,Gang Chen,Jian-wei Yin,Jin-xiang Dong

DOI: https://doi.org/10.1109/2.485845

2002-01-01

Journal of Zhejiang University SCIENCE A

Abstract:Constraint is an important aspect of role-based access control and is sometimes argued to be the principal motivation for role-based access control (RBAC). But so far few authors have discussed consistency maintenance for constraint in RBAC model. Based on researches of constraints among roles and types of inconsistency among constraints, this paper introduces corresponding formal rules, rule-based reasoning and corresponding methods to detect, avoid and resolve these inconsistencies. Finally, the paper introduces briefly the application of consistency maintenance in ZD-PDM, an enterprise-oriented product data management (FDM) system.
Constructingy Policies of Data Verifying Integrality in the Relational Database System

刘特,徐迎晓,李敏,何梅

DOI: https://doi.org/10.3969/j.issn.1000-3428.2002.11.034

2002-01-01

Abstract:Data from client is validated through constraints, rules, default and technique of trigger provided by the relational database system, thus validity and integrity of data source are ensured. The efficiency of data storage and search is improved by defining and deploying policy to verify data reasonably.

Integrity Constraints on Rich Data Types

Data Dependencies over Big Data: A Family Tree

Data Dependencies Extended for Variety and Veracity: A Family Tree

Data Dependencies Extended for Variety and Veracity: A Family Tree (extended Abstract)

Using Semantic Web Technologies to Specify Constraints of RBAC

Entity integrity management under data volume, variety and veracity

Integrity Constraints for Data Integration and Exchange Based on XML

Incorporating Integrity Constraints in Uncertain Databases

AUDITOR: A System Designed for Automatic Discovery of Complex Integrity Constraints in Relational Databases.

On Data Dependencies in Dataspaces

Constraints, Inference Channels and Secure Databases.

Integrity Constraints for General-Purpose Knowledge Bases

Embedded functional dependencies and data-completeness tailored database design

Introducing Keys and Integrity Rules in a Fuzzy Relational Data Model

Discovering Matching Dependencies

Detecting Data Errors with Statistical Constraints

Approximate Integrity Constraints in Incomplete Databases With Limited Domains

Data dependencies in the presence of difference

Comparable Dependencies over Heterogeneous Data.

Consistency Maintenance for Constraint in Role-Based Access Control Model

Constructingy Policies of Data Verifying Integrality in the Relational Database System