Abstract:The prevalence of chip multiprocessors opens opportunities of running data-parallel applications originally in clusters on a single machine with many cores. MapReduce, a simple and elegant programming model to program large-scale clusters, has recently been shown a promising alternative to harness the multicore platform. The differences such as memory hierarchy and communication patterns between clusters and multicore platforms raise new challenges to design and implement an efficient MapReduce system on multicore. This article argues that it is more efficient for MapReduce to iteratively process small chunks of data in turn than processing a large chunk of data at a time on shared memory multicore platforms. Based on the argument, we extend the general MapReduce programming model with a “tiling strategy”, called Tiled - MapReduce (TMR). TMR partitions a large MapReduce job into a number of small subjobs and iteratively processes one subjob at a time with efficient use of resources; TMR finally merges the results of all subjobs for output. Based on Tiled-MapReduce, we design and implement several optimizing techniques targeting multicore, including the reuse of the input buffer among subjobs, a NUCA/NUMA-aware scheduler, and pipelining a subjob’s reduce phase with the successive subjob’s map phase, to optimize the memory, cache, and CPU resources accordingly. Further, we demonstrate that Tiled-MapReduce supports fine-grained fault tolerance and enables several usage scenarios such as online and incremental computing on multicore machines. Performance evaluation with our prototype system called Ostrich on a 48-core machine shows that Ostrich saves up to 87.6% memory, causes less cache misses, and makes more efficient use of CPU cores, resulting in a speedup ranging from 1.86x to 3.07x over Phoenix. Ostrich also efficiently supports fine-grained fault tolerance, online, and incremental computing with small performance penalty.

Gmap Reduce: A Self-Adaption Mapreduce Framework Based on Granular Computing

A-MapCG: an Adaptive MapReduce Framework for GPUs.

An Analytical Performance Model of MapReduce

A MapReduce Computing Framework Based on GPU Cluster.

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Map-Balance-Reduce: an Improved Parallel Programming Model for Load Balancing of MapReduce.

GCPR:an Acceleration Method for PageRank Based on Graph-clustering on MapReduce

Research on the Analysis and Statistic of Geographical Conditions Based on the Strategy of "grid Index + MapReduce"

ActCap: Accelerating MapReduce on heterogeneous clusters with capability-aware data placement

The performance of MapReduce: an in-depth study

The Performance of MapReduce

LLMapReduce: Multi-Level Map-Reduce for High Performance Data Analysis

MAP-JOIN-REDUCE: Toward Scalable and Efficient Data Analysis on Large Clusters

Combination of in-memory graph computation with mapreduce: a subgraph-centric method of pagerank

AMREF: an Adaptive MapReduce Framework for Real Time Applications

An Advanced MapReduce: Cloud MapReduce, Enhancements and Applications

Tiled-Mapreduce: Optimizing Resource Usages Of Data-Parallel Applications On Multicore With Tiling

The Optimization of Memory Access Congestion for MapReduce Applications on Manycore Systems

Granular-ball computing: an efficient, robust, and interpretable adaptive multi-granularity representation and computation method

Tiled-MapReduce

Scalable Big Graph Processing in MapReduce