Abstract:MapReduce is a programming model and an associated implementation for processing and generating large data sets. Hadoop is an open-source implementation of Map Reduce, enjoying wide adoption, and is used not only for batch jobs but also for short jobs where low response time is critical. However, Hadoop's performance is currently limited by its default task scheduler, which implicitly assumes that cluster nodes are homogeneous and tasks make progress linearly, and uses these assumptions to decide when to speculatively re-execute tasks that appear to be stragglers. In practice, the homogeneity assumption does not always hold. Longest Approximate Time to End (LATE) is a Map Reduce scheduling algorithm that takes heterogeneous environments into consideration. It, however, adopts a static method to compute the progress of tasks. As a result neither Hadoop default nor LATE schedulers perform well in a heterogeneous environment. Self-adaptive Map Reduce Scheduling Algorithm (SAMR) uses historical information to adjust stage weights of map and reduce tasks when estimating task execution times. However, SAMR does not consider the fact that for different types of jobs their map and reduce stage weights may be different. Even for the same type of jobs, different datasets may lead to different weights. To this end, we propose ESAMR: an Enhanced Self-Adaptive Map Reduce scheduling algorithm to improve the speculative re-execution of slow tasks in Map Reduce. In ESAMR, in order to identify slow tasks accurately, we differentiate historical stage weights information on each node and divide them into k clusters using a k-means clustering algorithm and when executing a job's tasks on a node, ESAMR classifies the tasks into one of the clusters and uses the cluster's weights to estimate the execution time of the job's tasks on the node. Experimental results show that among the aforementioned algorithms, ESAMR leads to the smallest error in task execution time estimation and identifies slow tasks most accurately.

AMREF: an Adaptive MapReduce Framework for Real Time Applications

A task-level adaptive MapReduce framework for real-time streaming data in healthcare applications.

SAMR: A Self-adaptive MapReduce Scheduling Algorithm in Heterogeneous Environment.

ESAMR: An Enhanced Self-Adaptive MapReduce Scheduling Algorithm

The performance of MapReduce: an in-depth study

Progressive online aggregation in a distributed stream system

The Performance of MapReduce

An Adaptive Model For Building Service-Partition System

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Real-Time Scheduling in MapReduce Clusters

Adaptive memory reservation strategy for heavy workloads in the Spark environment

An Analytical Performance Model of MapReduce

An Adaptive Auto-configuration Tool for Hadoop

Self-Adaptive Map-Reduce Scheduling under Heterogeneous Environment

An Uncoupled Data Process and Transfer Model for MapReduce.

R-Store: A scalable distributed system for supporting real-time analytics

An Efficiency-Aware Scheduling for Data-Intensive Computations on Mapreduce Clusters

A-MapCG: an Adaptive MapReduce Framework for GPUs.

Design and Implementation of a Real-time Scheduling Algorithm for MapReduce

Adaptive Scheduling Framework of Streaming Applications based on Resource Demand Prediction with Hybrid Algorithms

On Traffic-Aware Partition and Aggregation in MapReduce for Big Data Applications