Abstract:Frequent itemset mining (FIM) is an important topic in data mining, which extracts knowledge of the relationships among items in a transaction dataset. Apriori algorithm and its variants, apriori-like algorithms, are widely used FIM algorithms. However, in a big data environment, these algorithms are inefficient. Due to the iterative calculation and modification of intermediate results, if an apriori-like algorithm is applied on a high-dimension or large-scale dataset, the memory requirement is unacceptable for a single machine. Although parallel and distributed programming could be a solution to deal with big data problems, apriori-like algorithms are not quite suitable for parallel computing because they need extra time overhead of communication to update intermediate results iteratively in cluster memories. To solve this problem, we propose a novel FIM algorithm, Distributed Apriori Based on Itemset-Encoding (DABIE). Different from existing methods, DABIE has two main advantages. Firstly, it stores intermediate results encoded in the form of 0 and 1 to reduce memory usage. Secondly, generating frequent itemsets is based on logical operation of encoding to reduce modification of data in cluster memories. These two advantages make DABIE more friendly to cluster computing. We apply DABIE on datasets with different scales. Compared with other distributed apriori-like algorithms, the results of our experiments show that DABIE can efficiently improve the multi-iterative FIM in big data environment.

Mining Frequent Item in Distributed Data Stream

Approximate mining of global closed frequent itemsets over data streams

A New Algorithm for Mining Global Frequent Itemsets in a Stream.

Gc-Tree: A Fast Online Algorithm For Mining Frequent Closed Itemsets

Mining Noise-Tolerant Frequent Closed Itemsets in Very Large Database.

A Novel Method to Generate Frequent Itemsets in Distributed Environment

Frequent Items Mining Based on Weight in Data Stream

Finding Frequent Closed Itemsets in Sliding Window in Linear Time.

Performance study of distributed Apriori-like frequent itemsets mining

DELAY: A Lazy Approach for Mining Frequent Patterns over High Speed Data Streams

YAFIM: A Parallel Frequent Itemset Mining Algorithm with Spark

An Algorithm of Mining Frequent Itemsets Based on Bloom Filter

Efficient Discovery of Emerging Frequent Patterns in ArbitraryWindows on Data Streams

Mining Robust Frequent Items in Data Streams

A STABLE PARALLEL DISTRIBUTED FREQUENT ITEMSET MINING ALGORITHM AND ITS APPLICATION

Finding frequent items in data streams using hierarchical information.

State-of-the-art on Frequent Pattern Mining in Data Streams

A Distributed Method for Fast Mining Frequent Patterns From Big Data

Estimation and maintenance of frequent pattern on data streams

Approximate Algorithm for Mining Frequent Items in Data Streams

Methods for Mining Frequent Items in Data Streams: an Overview