CREST: Towards Fast Speculation of Straggler Tasks in MapReduce

Lei,Tianyu Wo,Chunming Hu

DOI: https://doi.org/10.1109/icebe.2011.37

2011-01-01

Abstract:Data-Intensive Computing emerges as the fourth paradigm for modern scientific discoveries. MapReduce, a programming paradigm for large-scale data-parallel applications, is widely applied to web indexing, machine learning, and scientific simulations in industries as well as in academia. Recently, the virtualized "utility computing" environments, such as campus cloud, are becoming an important scenario to run MapReduce jobs. For a MapReduce job, the straggler tasks may dominate the response time and delay whole job. Various speculation schemes have been proposed to alleviate such problem, however, most of them implicitly assume that the time cost for data movement on launching speculative map tasks is trivial, which does not always hold for the virtualized Hadoop clusters in a campus cloud. In this paper, we propose a novel approach, CREST(Combination Re-Execution Scheduling Technology), which can achieve the optimal running time for speculative map tasks and decrease the response time of MapReduce jobs. The main idea is that re-executing a combination of tasks on a group of computing nodes may progress faster than directly speculating the straggler task on target node, due to data locality. The evaluation validates our approach and demonstrates that CREST can reduce the running time of a speculative map task by 70% with best cases and 50% on average, comparing with LATE.