Abstract:Complex data mining algorithms are processed in multiple iterations, where output of one iteration is used as input for the subsequent iterations. Existing parallel programming frameworks, e.g., MapReduce, Pregel and Spark, adopt the breadth first search (BFS) strategy to process those iterative jobs. They invoke the user-defined functions for every key-value pair or vertex to produce all possible intermediate results for the next iteration. Such BFS strategy incurs high I/O overheads, because normally, the size of intermediate search results of BFS is exponential to the size of original data, making it impossible to maintain those intermediate results in memory. In this paper, we present a new type of parallel programming model, the stack-centric model, where all computations are defined for a stack maintained in the distributed shared memory. The stack can be adaptively split into multiple stacks and disseminated to different compute nodes for parallel processing. The most distinguished feature of the stack-centric model is its support for the depth first search (DFS) algorithm which incurs much less memory overhead than its BFS counterpart. The maximal memory usage of DFS algorithm is determined by the height of its search tree, and hence, it is possible to conduct the computation of DFS algorithm mostly in memory. Our stack-centric model is not a pure DFS framework. It supports the hybrid BFS and DFS algorithms by tuning the trade-off between memory usage and parallelism. To show the advantages of stack-centric model, we implement two algorithms, frequent pattern mining algorithm and DNA sequence matching algorithm, on both stack-centric model and Spark. The memory usage of stack-centric model is 10 times less than the Spark, resulting in a significant performance improvement.

An Efficient Approach to Vectorize the Hybrid Breadth-First Search

Optimizing Data Accesses for Breadth-First Search on Shared Memory Computers.

An Adaptive Breadth-First Search Algorithm on Integrated Architectures

Highly Efficient Breadth-First Search on CPU-Based Single-Node System

Accelerating Breadth-First Graph Search on a Single Server by Dynamic Edge Trimming

Accelerating Graph-based Vector Search via Delayed-Synchronization Traversal

Energy-Efficient Graph Traversal on Integrated CPU-GPU Architectures

Designing and implementing a heuristic cross-architecture combination for graph traversal.

Parallel Cluster-BFS and Applications to Shortest Paths

FastBFS: Fast Breadth-First Graph Search on a Single Server.

Scalable Graph Traversal on Sunway TaihuLight with Ten Million Cores

Reducing Communication in Parallel Breadth-First Search on Distributed Memory Systems

An OpenMP‐based breadth‐first search implementation using the bag data structure

Optimizations to the Parallel Breath First Search on Distributed Memory

An Efficient Implementation of the Bellman-Ford Algorithm for Kepler GPU Architectures

Scaling Hop-Based Reachability Indexing for Fast Graph Pattern Query Processing

A study of vectorization for matrix-free finite element methods

Efficient and Effective Retrieval of Dense-Sparse Hybrid Vectors using Graph-based Approximate Nearest Neighbor Search

A Stack-Centric Processing Model for Iterative Processing

Memory-Constrained Vectorization and Scheduling of Dataflow Graphs for Hybrid CPU-GPU Platforms