Abstract:As large-scale data analytic becomes norm in various industries, using MapReduce frameworks to analyze ever-increasing volumes of data will keep growing. In turn, this trend drives up the intention to move MapReduce into multi-tenant clouds. However, the application performance of MapReduce can be significantly affected by the time-varying network bandwidth in a shared cluster. Although many recent studies improve MapReduce performance by dynamic scheduling to reduce the shuffle traffic, most of them do not consider the impact by widely existing hierarchical network architectures in data centers. In this article, we propose and design a hierarchical topology (Hit) aware MapReduce scheduler to minimize overall data traffic cost and hence to reduce job execution time. We first formulate the problem as a Topology Aware Assignment (TAA) optimization problem while considering dynamic computing and communication resources in the cloud with hierarchical network architecture. We further develop a synergistic strategy to solve the TAA problem by using the stable matching theory, which ensures the preference of both individual tasks and hosting machines. Finally, we implement the proposed scheduler as a pluggable module on Hadoop YARN and evaluate its performance by testbed experiments and simulations. The testbed experimental results show that Hit-scheduler can improve job completion time by 28 and 11 percent compared to Capacity Scheduler and Probabilistic Network-Aware scheduler, respectively. Our simulations further demonstrate that Hit-scheduler can reduce the traffic cost by 38 percent at most and the average shuffle flow traffic time by 32 percent compared to capacity scheduler. In this article, we have extended Hit-scheduler to a decentralized heuristic scheme to perform the policy-aware allocation in data center environments. Many existing centralized approximation approaches are too complex and infeasible to implement over a data center, which typically include lar-e amounts of servers, containers, switches, and traffic flows. In the extension, we have designed a decentralized heuristic scheme to perform the Policy-Aware Task (PAT) allocation by using existing centralize algorithm to approximately maximize the total gained utility. Finally, the simulation based experimental results show that the proposed PAT policy reduces the communication cost by 33.6 percent compared with the default scheduler in data centers.

HScheduler: an Optimal Approach to Minimize the Makespan of Multiple MapReduce Jobs

HMHS: Hybrid Multistage Heuristic Scheduling Algorithm for Heterogeneous MapReduce System

On Dynamic Job Ordering and Slot Configurations for Minimizing the Makespan Of Multiple MapReduce Jobs.

On Minimizing the Makespan of a Set of Offline MapReduce Jobs

A Note on “Orchestrating an Ensemble of MapReduce Jobs for Minimizing Their Makespan”

An Adaptive Scheduling Algorithm for Heterogeneous Hadoop Systems

Minimizing Makespan and Total Completion Time in MapReduce-like Systems

Heuristics for periodical batch job scheduling in a MapReduce computing framework

An Efficiency-Aware Scheduling for Data-Intensive Computations on Mapreduce Clusters

Online MapReduce Scheduling Problem of Minimizing the Makespan

An Efficient Job Scheduling for Map Reduce Clusters

Cluster Scheduler on Heterogeneous Cloud

Network-aware task selection to reduce multi-application makespan in cloud

An Optimized Algorithm for Reduce Task Scheduling

Joint Optimization of MapReduce Scheduling and Network Policy in Hierarchical Data Centers

Makespan Minimization for MapReduce Systems with Different Servers

Multi-search-routes-based Methods for Minimizing Makespan of Homogeneous and Heterogeneous Resources in Cloud Computing

Optimal Task Scheduling in MapReduce

Minimizing Total Busy Time in Offline Parallel Scheduling with Application to Energy Efficiency in Cloud Computing.

Hybrid Job-Driven Scheduling for Virtual MapReduce Clusters

Real-Time Scheduling in MapReduce Clusters