Abstract:Functional dependencies (FDs) play a very important role in many data management tasks such as schema normalization, data cleaning, and query optimization. Meanwhile, there are ever-increasing application demands for efficient FD discovery on large-scale datasets. Unfortunately, due to huge runtime and memory overhead, the existing single-machine FD discovery algorithms are inefficient for large-scale datasets. Recently, distributed data-parallel computing has become the de facto standard for large-scale data processing. However, it is challenging to design an efficient distributed FD discovery algorithm. In this paper, we present SmartFD, which is an efficient and scalable algorithm for distributed FD discovery. First, we propose a novel attribute sorting-based algorithm framework. Next, to discover all the FDs grouped by a given attribute, we propose an efficient distributed algorithm Attribute-centric Functional Dependency Discovery (AFDD). In AFDD, we design an Fast Sampling and Early Aggregation (FSEA) mechanism to improve the efficiency of distributed sampling and propose a memory-efficient index-based method for distributed FD validation. Moreover, AFDD employs an attribute-parallel method to accelerate the pruning-and-generation of candidate FDs. Furthermore, we propose an adaptive switching strategy between distributed sampling and distributed validation based on the unified time-based efficiency metric. Also, we employ a distributed probing based method to make the switching strategy more accurate. Experimental results on Apache Spark reveal that SmartFD outperforms the state-of-the-art single-machine algorithm HyFD and the existing distributed algorithm HFDD with 3.2 & x00D7;-44.9 & x00D7; and 2.5 & x00D7;-455.7 & x00D7; speedup respectively. Moreover, SmartFD achieves good row scalability and column scalability. Additionally, SmartFD has sub-linear node scalability.

Fuzzy DR Algorithm for Data Distribution Management

A fuzzy dead reckoning algorithm for distributed interactive applications

A fuzzy grouping mechanism for distributed interactive simulation.

Globally synchronized dead-reckoning with local lag for continuous distributed multiplayer games

Distributed Model Based on Data Partition and Load Balance Algorithm

Cluster Analysis And Fuzzy Query In Ship Maintenance And Design

Multi-resource Max-Min Fairness and Support Vector Machine Based DDoS Defense

Optimal control of DDoS defense with multi-resource max-min fairness

Data Management Solutions Based on the Data Distribution Service Communication Model

Drdda: Double Ruling Based Data Dissemination And Aggregation In Wireless Sensor Networks

Effective Data Distribution And Reallocation Strategies For Fast Query Response In Distributed Query-Intensive Data Environments

Concept Drift Detection Based on Decision Distribution in Inconsistent Information System

Dynamic Routing for Data Integrity and Delay Differentiated Services in Wireless Sensor Networks

Efficient and Scalable Functional Dependency Discovery on Distributed Data-Parallel Platforms.

Breaking Determinism: Fuzzy Modeling of Sequential Recommendation Using Discrete State Space Diffusion Model

A distributed parallel processing algorithm for FCD based on service bus

A Reinforcement-Learning-Based 3-D Estimation of Distribution Algorithm for Fuzzy Distributed Hybrid Flow-Shop Scheduling Considering On-Time-Delivery

Distributed Communication Connectivity Management Based On Fusion Data Quality

A Randomized Fuzzy Load Balancing Algorithm for Distributed Streaming Systems

A Parallel Matching Algorithm Based on Order Relation for HLA Data Distribution Management

Distributed Demand Response for Multienergy Residential Communities With Incomplete Information