Abstract:Order dependencies (ODs) are recently proposed to describe a relationship of ordering between lists of attributes. It is typically too costly to design ODs manually, since the number of possible ODs is of a factorial complexity in the number of attributes. To this end, automatic discovery techniques for ODs are developed. In practice, data is frequently updated, especially with tuple insertions. Existing techniques do not lend themselves well to these situations, since it is prohibitively expensive to recompute all ODs from scratch after every update. In this paper, we make a first effort to investigate incremental OD discovery techniques in response to tuple insertions. Given a relation D, a set \(\varSigma \) of valid and minimal ODs on D, and a set \(\triangle D\) of tuple insertions to D, it is to find, changes \(\triangle \varSigma \) to \(\varSigma \) that makes \(\varSigma \oplus \triangle \varSigma \) a set of valid and minimal ODs on \(D + \triangle D\). Note that \(\triangle \varSigma \) contains both new ODs to be added to \(\varSigma \) and outdated ODs to be removed from \(\varSigma \). Specifically, (1) We formalize the incremental OD discovery problem. Although the incremental discovery problem has a same complexity as its batch (non-incremental) counterpart in terms of traditional complexity, we show that it has good data locality. It is linear in the size of \(\triangle D\) to validate on \(D+\triangle D\) any OD \(\varphi \) that is valid on D. (2) We present effective incremental OD discovery techniques, leveraging an intelligent traversal strategy for finding \(\triangle \varSigma \) and chosen indexes to minimize access to D. Our approach computes \(\triangle \varSigma \) based on ODs in \(\varSigma \), and is independent of the size of D. (3) Using real-life data, we experimentally verify that our approach substantially outperforms its batch counterpart by orders of magnitude.

Efficient Set-Based Order Dependency Discovery with a Level-Wise Hybrid Strategy

Approximate Order Dependency Discovery

Efficient Bidirectional Order Dependency Discovery

Discovery of Approximate Lexicographical Order Dependencies

Incremental Discovery of Order Dependencies on Tuple Insertions

Repairing Data Violations With Order Dependencies

Dynamic Functional Dependency Discovery with Dynamic Hitting Set Enumeration

Discovering Approximate Functional Dependencies from Distributed Big Data.

Fast incremental discovery of pointwise order dependencies

Functional Dependencies Discovering in Distributed Big Data

Efficient Differential Dependency Discovery

Fast Algorithm for Embedded Order Dependency Validation (Extended Version)

Discovering Domain Orders through Order Dependencies

Efficient Relaxed Functional Dependency Discovery with Minimal Set Cover

A Hybrid Approach Combining R*-Tree and k-d Trees to Improve Linked Open Data Query Performance

OPTIMIZED ALGORITHM OF DISCOVERING FUNCTIONAL DEPENDENCIES WITH DEGREES OF SATISFACTION

Efficient and Scalable Functional Dependency Discovery on Distributed Data-Parallel Platforms.

Efficient Discovery of Functional Dependencies on Massive Data

Scalable Functional Dependencies Discovery from Big Data

Inconsistency Detection in Distributed Big Data

Relaxed Functional Dependency Discovery in Heterogeneous Data Lakes