Abstract:Complex data mining algorithms are processed in multiple iterations, where output of one iteration is used as input for the subsequent iterations. Existing parallel programming frameworks, e.g., MapReduce, Pregel and Spark, adopt the breadth first search (BFS) strategy to process those iterative jobs. They invoke the user-defined functions for every key-value pair or vertex to produce all possible intermediate results for the next iteration. Such BFS strategy incurs high I/O overheads, because normally, the size of intermediate search results of BFS is exponential to the size of original data, making it impossible to maintain those intermediate results in memory. In this paper, we present a new type of parallel programming model, the stack-centric model, where all computations are defined for a stack maintained in the distributed shared memory. The stack can be adaptively split into multiple stacks and disseminated to different compute nodes for parallel processing. The most distinguished feature of the stack-centric model is its support for the depth first search (DFS) algorithm which incurs much less memory overhead than its BFS counterpart. The maximal memory usage of DFS algorithm is determined by the height of its search tree, and hence, it is possible to conduct the computation of DFS algorithm mostly in memory. Our stack-centric model is not a pure DFS framework. It supports the hybrid BFS and DFS algorithms by tuning the trade-off between memory usage and parallelism. To show the advantages of stack-centric model, we implement two algorithms, frequent pattern mining algorithm and DNA sequence matching algorithm, on both stack-centric model and Spark. The memory usage of stack-centric model is 10 times less than the Spark, resulting in a significant performance improvement.

Combining Distributed Classifies by Stacking

Enhancing binary classification: A new stacking method via leveraging computational geometry

Geographic stacking: Decision fusion to increase global land cover map accuracy

A boosted clustering algorithm for distributed homogeneous data mining

A Stack-Centric Processing Model for Iterative Processing

Stacking Learning with Coalesced Cost Filtering for Accurate Stereo Matching

Reranking For Stacking Ensemble Learning

Dynamic Classifier Selection for Effective Mining from Noisy Data Streams

Exploration of the Stacking Ensemble Machine Learning Algorithm for Cheating Detection in Large-Scale Assessment

Multiclass Classification Based On Meta Probability Codes

Combining multiple statistical classifiers to improve the accuracy of task classification

Stacking approach for accurate Invasive Ductal Carcinoma classification

Multi-classification assessment of personal credit risk based on stacking integration

Distributed Model Reuse with Multiple Classifiers

Stacked Generalization: when Does It Work?

Combination of Multiple Classifier Using Feature Space Partitioning

Dynamic Stacked Generalization for Node Classification on Networks

Semi-Stacking for Semi-supervised Sentiment Classification.

Stacking Ensemble Technique for Classifying Breast Cancer

Using Correspondence Analysis to Combine Classifiers

Evaluating StackingC and ensemble models for enhanced lithological classification in geological mapping