Abstract:MapReduce is a widely used parallel computing framework for large scale data processing. The two major performance metrics in MapReduce are job execution time and cluster throughput. They can be seriously impacted by straggler machines—machines on which tasks take an unusually long time to finish. Speculative execution is a common approach for dealing with the straggler problem by simply backing up those slow running tasks on alternative machines. Multiple speculative execution strategies have been proposed, but they have some pitfalls: i) Use average progress rate to identify slow tasks while in reality the progress rate can be unstable and misleading, ii) Cannot appropriately handle the situation when there exists data skew among the tasks, iii) Do not consider whether backup tasks can finish earlier when choosing backup worker nodes. In this paper, we first present a detailed analysis of scenarios where existing strategies cannot work well. Then we develop a new strategy, maximum cost performance (MCP), which improves the effectiveness of speculative execution significantly. To accurately and promptly identify stragglers, we provide the following methods in MCP: i) Use both the progress rate and the process bandwidth within a phase to select slow tasks, ii) Use exponentially weighted moving average (EWMA) to predict process speed and calculate a task’s remaining time, iii) Determine which task to backup based on the load of a cluster using a cost-benefit model. To choose proper worker nodes for backup tasks, we take both data locality and data skew into consideration. We evaluate MCP in a cluster of 101 virtual machines running a variety of applications on 30 physical servers. Experiment results show that MCP can run jobs up to 39 percent faster and improve the cluster throughput by up to 44 percent compared to Hadoop-0.21.

High-integrity mapreduce computation in cloud with speculative execution

Improving MapReduce Performance with Partial Speculative Execution

Dominoes: Speculative Repair in Erasure-Coded Hadoop System.

Practical Verifiable Computation–A MapReduce Case Study

MtMR: Ensuring MapReduce Computation Integrity with Merkle Tree-Based Verifications

A Survey Of Speculative Execution Strategy In Mapreduce

Shiyf: A Secured And High-Integrity Yarn Framework

Practical Verification of MapReduce Computation Integrity via Partial Re-execution

Enhancing MapReduce Fault Recovery Through Binocular Speculation

Improving MapReduce Performance Using Smart Speculative Execution Strategy

Moving Hadoop into the Cloud with Flexible Slot Management and Speculative Execution

Security and Privacy Aspects in MapReduce on Clouds: A Survey

Esplash: Efficient Speculation in Large Scale Heterogeneous Computing Systems

An Incremental Partitioning Strategy for Data Balance on MapReduce

CREST: Towards Fast Speculation of Straggler Tasks in MapReduce

An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in A Multi- Job Heterogeneous Environment

Improving MapReduce Performance in a Heterogeneous Cloud: A Measurement Study

Location-Aware Data Block Allocation Strategy for HDFS-Based Applications in the Cloud

An Optimized Speculative Execution Strategy Based on Local Data Prediction in a Heterogeneous Hadoop Environment

Halt or continue: estimating progress of queries in the cloud

Near-data Prediction Based Speculative Optimization in a Distribution Environment