Abstract:MapReduce is a programming model and an associated implementation for processing and generating large data sets. Hadoop is an open-source implementation of Map Reduce, enjoying wide adoption, and is used not only for batch jobs but also for short jobs where low response time is critical. However, Hadoop's performance is currently limited by its default task scheduler, which implicitly assumes that cluster nodes are homogeneous and tasks make progress linearly, and uses these assumptions to decide when to speculatively re-execute tasks that appear to be stragglers. In practice, the homogeneity assumption does not always hold. Longest Approximate Time to End (LATE) is a Map Reduce scheduling algorithm that takes heterogeneous environments into consideration. It, however, adopts a static method to compute the progress of tasks. As a result neither Hadoop default nor LATE schedulers perform well in a heterogeneous environment. Self-adaptive Map Reduce Scheduling Algorithm (SAMR) uses historical information to adjust stage weights of map and reduce tasks when estimating task execution times. However, SAMR does not consider the fact that for different types of jobs their map and reduce stage weights may be different. Even for the same type of jobs, different datasets may lead to different weights. To this end, we propose ESAMR: an Enhanced Self-Adaptive Map Reduce scheduling algorithm to improve the speculative re-execution of slow tasks in Map Reduce. In ESAMR, in order to identify slow tasks accurately, we differentiate historical stage weights information on each node and divide them into k clusters using a k-means clustering algorithm and when executing a job's tasks on a node, ESAMR classifies the tasks into one of the clusters and uses the cluster's weights to estimate the execution time of the job's tasks on the node. Experimental results show that among the aforementioned algorithms, ESAMR leads to the smallest error in task execution time estimation and identifies slow tasks most accurately.

Self-Adaptive Map-Reduce Scheduling under Heterogeneous Environment

SAMR: A Self-adaptive MapReduce Scheduling Algorithm in Heterogeneous Environment.

HAT: history-based auto-tuning MapReduce in heterogeneous environments

ESAMR: An Enhanced Self-Adaptive MapReduce Scheduling Algorithm

An Efficient Fault-Tolerant Scheduling Algorithm for Periodic Real-Time Tasks in Heterogeneous Platforms

An Efficiency-Aware Scheduling for Data-Intensive Computations on Mapreduce Clusters

Real-Time Scheduling in MapReduce Clusters

Survey on MapReduce Scheduling Algorithms

HMHS: Hybrid Multistage Heuristic Scheduling Algorithm for Heterogeneous MapReduce System

A Parameter Dynamic-Tuning Scheduling Algorithm Based on History in Heterogeneous Environments.

A Delay Scheduling Algorithm Based on History Time in Heterogeneous Environments

Matchmaking: A New MapReduce Scheduling Technique

Design and Implementation of a Real-time Scheduling Algorithm for MapReduce

A Dispatching-Rule-Based Task Scheduling Policy for MapReduce with Multi-type Jobs in Heterogeneous Environments

K%-Fair Scheduling: A Flexible Task Scheduling Strategy for Balancing Fairness and Efficiency in MapReduce Systems

Adaptive Task Scheduling Strategy Based on Dynamic Workload Adjustment for Heterogeneous Hadoop Clusters

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Research of Scheduling Strategy Based on Fault Tolerance in Hadoop Platform

An Locality-Aware Scheduling Based on a Novel Scheduling Model to Improve System Throughput of MapReduce Cluster

Optimizing Internal Overlaps by Self-Adjusting Resource Allocation in Multi-Stage Computing Systems

Delay Scheduling Algorithm Based on Task Classification