Abstract:Traditional approaches to data mining are generally designed for small, centralized, and static datasets. However, when a dataset grows at an enormous rate, the algorithms become infeasible in terms of huge consumption of computational and I/O resources. Frequent itemset mining (FIM) is one of the key algorithms in data mining and finds applications in a variety of domains; however, traditional algorithms do face problems in efficiently processing large and dynamic datasets. This research introduces a distributed incremental approximation frequent itemset mining (DIAFM) algorithm that tackles the mentioned challenges using shard-based approximation within the MapReduce framework. DIAFM minimizes the computational overhead of a program by reducing dataset scans, bypassing exact support checks, and incorporating shard-level error thresholds for an appropriate trade-off between efficiency and accuracy. Extensive experiments have demonstrated that DIAFM reduces runtime by 40–60% compared to traditional methods with losses in accuracy within 1–5%, even for datasets over 500,000 transactions. Its incremental nature ensures that new data increments are handled efficiently without needing to reprocess the entire dataset, making it particularly suitable for real-time, large-scale applications such as transaction analysis and IoT data streams. These results demonstrate the scalability, robustness, and practical applicability of DIAFM and establish it as a competitive and efficient solution for mining frequent itemsets in distributed, dynamic environments.

A Novel Incremental Algorithm For Mining Frequent Itemsets

Mining Noise-Tolerant Frequent Closed Itemsets in Very Large Database.

A New Algorithm for Mining Global Frequent Itemsets in a Stream.

Incremental Updating Algorithm for Mining Maximum Frequent Itemsets

An Incremental Algorithm For Mining Privacy-Preserving Frequent Itemsets

Gc-Tree: A Fast Online Algorithm For Mining Frequent Closed Itemsets

An Incremental Algorithm For Frequent Itemset Mining On Spark

A decremental algorithm for maintaining frequent itemsets in dynamic databases

A Decremental Algorithm of Frequent Itemset Maintenance for Mining Updated Databases

Fast Updating Frequent Itemsets Based on Classifying Incremental Transactions

A Fast Algorithm for Maintenance of Association Rules in Incremental Databases

Efficiently Mining Frequent Itemsets on Massive Data

SA-IFIM: incrementally mining frequent itemsets in update distorted databases

Incremental Updating Algorithm of Frequent Itemsets Based on FP_tree

A Spark-based Incremental Algorithm for Frequent Itemset Mining

DIAFM: An Improved and Novel Approach for Incremental Frequent Itemset Mining

An Improved Mining Algorithm of Maximal Frequent Itemsets

IFCIA: an Efficient Algorithm for Mining Intertransaction Frequent Closed Itemsets

Incremental Mining Large Itemsets with Constraints in Dynamic Databases

Mining recent frequent itemsets in data streams by radioactively attenuating strategy

Updating Algorithm for Association Rules Based on Fully Mining Incremental Transactions