Abstract:Complex data mining algorithms are processed in multiple iterations, where output of one iteration is used as input for the subsequent iterations. Existing parallel programming frameworks, e.g., MapReduce, Pregel and Spark, adopt the breadth first search (BFS) strategy to process those iterative jobs. They invoke the user-defined functions for every key-value pair or vertex to produce all possible intermediate results for the next iteration. Such BFS strategy incurs high I/O overheads, because normally, the size of intermediate search results of BFS is exponential to the size of original data, making it impossible to maintain those intermediate results in memory. In this paper, we present a new type of parallel programming model, the stack-centric model, where all computations are defined for a stack maintained in the distributed shared memory. The stack can be adaptively split into multiple stacks and disseminated to different compute nodes for parallel processing. The most distinguished feature of the stack-centric model is its support for the depth first search (DFS) algorithm which incurs much less memory overhead than its BFS counterpart. The maximal memory usage of DFS algorithm is determined by the height of its search tree, and hence, it is possible to conduct the computation of DFS algorithm mostly in memory. Our stack-centric model is not a pure DFS framework. It supports the hybrid BFS and DFS algorithms by tuning the trade-off between memory usage and parallelism. To show the advantages of stack-centric model, we implement two algorithms, frequent pattern mining algorithm and DNA sequence matching algorithm, on both stack-centric model and Spark. The memory usage of stack-centric model is 10 times less than the Spark, resulting in a significant performance improvement.

Combination of in-memory graph computation with mapreduce: a subgraph-centric method of pagerank

Evaluating Large Graph Processing in MapReduce Based on Message Passing

GCPR:an Acceleration Method for PageRank Based on Graph-clustering on MapReduce

Exploring Computation Locality of Graph Mining Algorithms on MapReduce

Online Multi-keywords Map-search Algorithm Based on Sub-region Method

Efficiently extracting frequent subgraphs using MapReduce

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Tuning the granularity of parallelism for distributed graph processing

A parallel graph partitioning algorithm to speed up the large-scale distributed graph mining.

LI-MR: A Local Iteration Map/Reduce Model and Its Application to Mine Community Structure in Large-Scale Networks

A Parallel Computing Model for Large-Graph Mining with MapReduce.

Distributed PageRank Computation Based on Iterative Aggregation-Disaggregation Methods.

LightGraph: Lighten Communication in Distributed Graph-Parallel Processing

Locality Sensitive Hashing for Optimizing Subgraph Query Processing in Parallel Computing Systems.

Efficient Query Evaluation on Distributed Graphs with Hadoop Environment

A Stack-Centric Processing Model for Iterative Processing

Research on the Analysis and Statistic of Geographical Conditions Based on the Strategy of "grid Index + MapReduce"

Community structure mining in big data social media networks with MapReduce

Parallelized Similarity Flooding Algorithm for Processing Large Scale Graph Datasets with MapReduce

Towards Efficient Subgraph Search In Cloud Computing Environments

An Optimal Locality-Aware Task Scheduling Algorithm Based on Bipartite Graph Modelling for Spark Applications