Abstract:Functional dependencies (FDs) play a very important role in many data management tasks such as schema normalization, data cleaning, and query optimization. Meanwhile, there are ever-increasing application demands for efficient FD discovery on large-scale datasets. Unfortunately, due to huge runtime and memory overhead, the existing single-machine FD discovery algorithms are inefficient for large-scale datasets. Recently, distributed data-parallel computing has become the de facto standard for large-scale data processing. However, it is challenging to design an efficient distributed FD discovery algorithm. In this paper, we present SmartFD, which is an efficient and scalable algorithm for distributed FD discovery. First, we propose a novel attribute sorting-based algorithm framework. Next, to discover all the FDs grouped by a given attribute, we propose an efficient distributed algorithm Attribute-centric Functional Dependency Discovery (AFDD). In AFDD, we design an Fast Sampling and Early Aggregation (FSEA) mechanism to improve the efficiency of distributed sampling and propose a memory-efficient index-based method for distributed FD validation. Moreover, AFDD employs an attribute-parallel method to accelerate the pruning-and-generation of candidate FDs. Furthermore, we propose an adaptive switching strategy between distributed sampling and distributed validation based on the unified time-based efficiency metric. Also, we employ a distributed probing based method to make the switching strategy more accurate. Experimental results on Apache Spark reveal that SmartFD outperforms the state-of-the-art single-machine algorithm HyFD and the existing distributed algorithm HFDD with 3.2 & x00D7;-44.9 & x00D7; and 2.5 & x00D7;-455.7 & x00D7; speedup respectively. Moreover, SmartFD achieves good row scalability and column scalability. Additionally, SmartFD has sub-linear node scalability.

Mining Conditional Functional Dependency Rules on Big Data

Mining CFD Rules on Big Data

Fuzzy Clustering-Based Quantitative Association Rules Mining in Multidimensional Data Set

Mining Fuzzy Association Rules in Data Streams

Guided conditional functional dependency discovery

Dtgc-Tree: A New Strategy Of Association Rules Mining

Using Conditional Functional Dependency to Discover Abnormal Data in RDF Graphs.

Association Rules Mining Based on the Discriminative Concept Lattice

Mining Noise-Tolerant Frequent Closed Itemsets in Very Large Database.

Scalable Functional Dependencies Discovery from Big Data

Constraint-Adaptive Rule Mining in Large Databases

DAFDiscover: Robust Mining Algorithm for Dynamic Approximate Functional Dependencies on Dirty Data

Conditional Regression Rules

Efficient Relaxed Functional Dependency Discovery with Minimal Set Cover

Analyses and Validation of Conditional Dependencies with Built-in Predicates

Data Dependencies over Big Data: A Family Tree

Properties and Pre-Processing Strategies to Enhance the Discovery of Functional Dependency with Degree of Satisfaction

EulerFD: an Efficient Double-Cycle Approximation of Functional Dependencies

Discovering Abnormal Data in RDF Knowledge Base

Efficient and Scalable Functional Dependency Discovery on Distributed Data-Parallel Platforms.

Efficient Discovery of Functional Dependencies with Degrees of Satisfaction.