Abstract:• A task-parallel framework for local structural vertex similarity calculation. • A threshold-adaptive technique to execute a similarity calculation task. • A Spark-based implementation with three optimization techniques. • Comprehensive performance evaluation of the vertex similarity calculation strategies. In many graph analytical applications, the local structural vertex similarity calculation is an essential prerequisite for advanced graph mining. The similarity calculation finds out all the similar vertex pairs whose local structural similarity scores (like the number of common neighbors, and the Jaccard index of adjacency sets) are above a given threshold. The real-world applications use a wide range of similarity thresholds. However, the existing distributed methods for the problem only optimize for either high thresholds (> 0.7) or low thresholds (< 0.1). To overcome the drawback, we propose a new distributed vertex similarity calculation framework VSIM that is efficient under a broad range of thresholds. VSIM processes static undirected graphs with local structural similarity scores that measure the similarity between vertices based on the first-order topology information. VSIM generates a similarity calculation task for every vertex in the graph and conducts all the tasks in parallel on a distributed computing platform along with a distributed key-value store. Each task finds vertices similar to a given center vertex with two task execution modes. The two modes optimize for high and low thresholds, respectively. Each task picks the suitable mode adaptively according to cost estimation models. We also propose an efficient implementation for VSIM on Apache Spark, with three optimization techniques to reduce communication costs and balance workloads on power-law graphs. The experimental evaluation shows that VSIM outperforms the state-of-the-art distributed methods by up to 67x speedup. VSIM can achieve near-linear node scalability in low-threshold and small cache scenarios.

Parallelized Similarity Flooding Algorithm for Processing Large Scale Graph Datasets with MapReduce

Distributed Affinity Propagation Clustering Based on MapReduce

Evaluating Large Graph Processing in MapReduce Based on Message Passing

A Parallel Computing Model for Large-Graph Mining with MapReduce.

Distributed structural clustering on large graph

Efficiently extracting frequent subgraphs using MapReduce

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

Combination of in-memory graph computation with mapreduce: a subgraph-centric method of pagerank

A Parallel Algorithm for Erosion Slope Length Calculation Based on Grid Terrain Datasets

VSIM: Distributed Local Structural Vertex Similarity Calculation on Big Graphs

Towards Distributed Node Similarity Search on Graphs

A parallel graph partitioning algorithm to speed up the large-scale distributed graph mining.

An Efficient MapReduce Algorithm for Similarity Join in Metric Spaces

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

A Feasible Graph Partition Framework for Parallel Computing of Big Graph

Parallel Algorithms for Extracting Drainage Network from Digital Elevation Models

Scaling Hop-Based Reachability Indexing for Fast Graph Pattern Query Processing

Large-Scale Social Network Analysis Based on MapReduce

Research on Large Scale Parallel Hydrological Simulation

Scalable Parallel Join for Huge Tables

Parallel and distributed architecture of genetic algorithm on Apache Hadoop and Spark