Abstract:Complex data mining algorithms are processed in multiple iterations, where output of one iteration is used as input for the subsequent iterations. Existing parallel programming frameworks, e.g., MapReduce, Pregel and Spark, adopt the breadth first search (BFS) strategy to process those iterative jobs. They invoke the user-defined functions for every key-value pair or vertex to produce all possible intermediate results for the next iteration. Such BFS strategy incurs high I/O overheads, because normally, the size of intermediate search results of BFS is exponential to the size of original data, making it impossible to maintain those intermediate results in memory. In this paper, we present a new type of parallel programming model, the stack-centric model, where all computations are defined for a stack maintained in the distributed shared memory. The stack can be adaptively split into multiple stacks and disseminated to different compute nodes for parallel processing. The most distinguished feature of the stack-centric model is its support for the depth first search (DFS) algorithm which incurs much less memory overhead than its BFS counterpart. The maximal memory usage of DFS algorithm is determined by the height of its search tree, and hence, it is possible to conduct the computation of DFS algorithm mostly in memory. Our stack-centric model is not a pure DFS framework. It supports the hybrid BFS and DFS algorithms by tuning the trade-off between memory usage and parallelism. To show the advantages of stack-centric model, we implement two algorithms, frequent pattern mining algorithm and DNA sequence matching algorithm, on both stack-centric model and Spark. The memory usage of stack-centric model is 10 times less than the Spark, resulting in a significant performance improvement.

Pre-stack Kirchhoff Time Migration on Hadoop and Spark.

Distributed High-Dimension Matrix Operation Optimization on Spark

Implementation strategy of 3D reverse time migration on GPU/CPU clusters

Parallel Algorithm for Pre-Stack Depth Migration Based on Data Attributes and Data Serving

Generalized Screen Method for Prestack Time Migration

Prestack Depth Migration Method Based on Quasi-Linear Born Approximation

A Stack-Centric Processing Model for Iterative Processing

Spark-based intelligent parameter inversion method for prestack seismic data

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Domain-specific Pattern Matching Based Automatic Parallelization: Demonstrated by 2-D Prestack Migration

An efficient 3D wave-equation pre-stack time migration for high-density and wide-azimuth data

Towards Efficient Reverse-time Migration Imaging Computation by Pipeline and Fine-grained Execution Parallelization.

Comparison of Kirchhoff and reverse‐time migration methods with applications to prestack depth imaging of complex structures

Performance Analysis and Optimization of a Hybrid Distributed Reverse Time Migration Application

Real-Time Spatial Queries for Moving Objects Using Storm Topology.

Optimizing and accelerating space-time Ripley's K function based on Apache Spark for distributed spatiotemporal point pattern analysis

Building Near-Real-Time Processing Pipelines with the Spark-MPI Platform

High-Resolution Anisotropic Prestack Kirchhoff Dynamic Focused Beam Migration

Revisiting Finite Difference and Spectral Migration Methods on Diverse Parallel Architectures

Distributed Streaming Analytics on Large-scale Oceanographic Data using Apache Spark

Efficient Spatiotemporal Interpolation with Spark Machine Learning