Abstract:With the development of information technology and internet, various types of information are increasing explosively. It is still a challenge to discover knowledge from massive information. As a pivotal technology to obtain knowledge, data mining has attracted a large amount of research interest for several decades; however, when dealing with large-scale data, most of previous works are still not as efficient as expected. Therefore, the extension of algorithms to deal with large-scale data and the improvement of executing efficiency have become important issues in data mining. Cloud computing based data mining has become a hot topic recently. In this paper, we develop a parallel and distributed data mining toolkit platform (PDMiner) based on large-scale data processing platform—Hadoop. In PDMiner, we propose to implement various data mining operations, such as data preprocessing, association rule analysis, classification and clustering in a parallel manner. The experimental results show that these parallel algorithms 1) can tackle large-scale data set, up to terabyte; 2) are very high efficiency, since they have good speedup; 3) are easily extended to execute in a cluster of commodity machines, which can make full use of computing resource; 4) are efficient for practical data mining. Additionally, we develop knowledge flow subsystem, which can facilitate the user to define data mining task in PDMiner. Furthermore, we can conveniently integrate new parallel algorithms into PDMiner through flexible interface.

Research on Parallel Mining Frequent Patterns with Taboo Constraints

Parallel Frequent Pattern Discovery: Challenges and Methodology

Research on Parallel Association Rules Mining Algorithm Based on Hadoop

Distributed Mining of Frequent Patterns in Big Data by Hybrid Strategies.

Research on Association Rules Mining Algorithm Based on Hadoop-Taking Apriori as an Example

An Incremental Updating Fp_tree Algorithm On Hadoop

Tree Partition Based Parallel Frequent Pattern Mining on Shared Memory Systems

Study of Improved Algorithm for Mining Based on Frequent Pattern Array

A Mining Algorithm for Frequent Patterns Based on Prefix Tree

State-of-the-art on Frequent Pattern Mining in Data Streams

Mining Method of Frequent Patterns Based on Temporal Constraints

Parallel mining and application of fuzzy association rules

Parallel Apriori Algorithm Based on the Thread Pool

Investigation of Parallel Data Mining

Parallel Data Mining Optimal Algorithm of Virtual Cluster

PDMiner: a cloud computing based parallel and distributed data mining toolkit platform

Mining Frequent Induced Subtree Patterns with Subtree-Constraint

A MapReduce-Based Parallel Frequent Pattern Growth Algorithm for Spatiotemporal Association Analysis of Mobile Trajectory Big Data

A Novel Parallel Algorithm for Frequent Itemsets Mining in Massive Small Files Datasets

Mining Project Frequent Patterns without Candidate Generation

Preference-Based Frequent Pattern Mining