Abstract:In this paper, we present WOHA, an efficient scheduling framework for deadline-aware Map-Reduce workflows. In data centers, complex backend data analysis often utilizes a workflow that contains tens or even hundreds of interdependent Map-Reduce jobs. Meeting deadlines of these workflows is usually of crucial importance to businesses (for example, workflows tightly linked to time-sensitive advertisement placement optimizations can directly affect revenue). Popular Map-Reduce implementations, such as Hadoop, deal with independent Map-Reduce jobs rather than workflows of jobs. In order to simplify the process of submitting workflows, solutions like Oozie emerge, which take a workflow configuration file as input and automatically submit its Hadoop jobs at the right time. The information separation that Hadoop only handles resource allocation and Oozie workflow topology, although preventing the Hadoop master node from getting involved with complex workflow analysis, may unnecessarily lengthen the workflow spans and thus cause more deadline misses. To address this problem and at the same time honor the efficiency of Hadoop master node, WOHA allows client nodes to locally generate scheduling plans which are later used as resource allocation hints by the master node. Under this framework design, we propose a novel scheduling algorithm that improves deadline satisfaction ratio by dynamically assigning priorities among workflows based on their progresses. We implement WOHA by extending Hadoop-1.2.1. Our experiments over an 80-server cluster show that WOHA manages to increase the deadline satisfaction ratio by 10% compared to state-of-the-art solutions, and scales up to tens of thousands of concurrently running workflows.

Research on the Framework and Resource Scheduling Mechanisms of Hadoop YARN

New Scheduling Algorithms for Improving Performance and Resource Utilization in Hadoop YARN Clusters

On Global Resource Allocation in Clusters for Data Analytics.

A Novel Job Scheduling Model to Enhance Efficiency and Overall User Fairness of Cloud Computing Environment.

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Application-adaptive Resource Scheduling in a Computational Grid

Performance evaluation of job schedulers on Hadoop YARN

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Multiple Resources Scheduling for Diverse Workloads in Heterogeneous Datacenter

Push-based Network-efficient Hadoop YARN Scheduling Mechanism for In-memory Computing

Improving Scheduling Efficiency of Hadoop YARN Using AFSA Algorithm.

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds.

Multi-policy-aware MapReduce Resource Allocation and Scheduling for Smart Computing Cluster.

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds

Performance optimization of computing task scheduling based on the Hadoop big data platform

SAF: Simulated Annealing Fair Scheduling for Hadoop Yarn Clusters

Secapacity: A Secure Capacity Scheduler In Yarn

Hadoop Scheduling Base On Data Locality

WOHA: Deadline-Aware Map-Reduce Workflow Scheduling Framework over Hadoop Clusters

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Research on Embedded Real-time System Scheduling Framework