Abstract:In this paper, we present WOHA, an efficient scheduling framework for deadline-aware Map-Reduce workflows. In data centers, complex backend data analysis often utilizes a workflow that contains tens or even hundreds of interdependent Map-Reduce jobs. Meeting deadlines of these workflows is usually of crucial importance to businesses (for example, workflows tightly linked to time-sensitive advertisement placement optimizations can directly affect revenue). Popular Map-Reduce implementations, such as Hadoop, deal with independent Map-Reduce jobs rather than workflows of jobs. In order to simplify the process of submitting workflows, solutions like Oozie emerge, which take a workflow configuration file as input and automatically submit its Hadoop jobs at the right time. The information separation that Hadoop only handles resource allocation and Oozie workflow topology, although preventing the Hadoop master node from getting involved with complex workflow analysis, may unnecessarily lengthen the workflow spans and thus cause more deadline misses. To address this problem and at the same time honor the efficiency of Hadoop master node, WOHA allows client nodes to locally generate scheduling plans which are later used as resource allocation hints by the master node. Under this framework design, we propose a novel scheduling algorithm that improves deadline satisfaction ratio by dynamically assigning priorities among workflows based on their progresses. We implement WOHA by extending Hadoop-1.2.1. Our experiments over an 80-server cluster show that WOHA manages to increase the deadline satisfaction ratio by 10% compared to state-of-the-art solutions, and scales up to tens of thousands of concurrently running workflows.

ACO Algorithm-based Parallel Job Scheduling Investigation on Hadoop

An Ant Colony Optimization Approach for No-Wait Flow-Line Batch Scheduling with Limited Batch Sizes

A Deadline And Budget Constrained Cost-Time Optimization Algorithm For Scheduling Dependent Tasks In Grid Computing

Performance optimization of computing task scheduling based on the Hadoop big data platform

Hybrid Hierarchical Particle Swarm Optimization with Evolutionary Artificial Bee Colony Algorithm for Task Scheduling in Cloud Computing

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Study on the Convergence of Converse Ant Colony Algorithm for Job Shop Scheduling Problem.

Effect of copper(II) chloride on suppression of racemization in peptide synthesis by the mixed anhydride and related methods.

The Improved Job Scheduling Algorithm of Hadoop Platform

Improving Scheduling Efficiency of Hadoop YARN Using AFSA Algorithm.

Task scheduling in cloud computing systems using honey badger algorithm with improved density factor and foucault pendulum motion

WOHA: Deadline-Aware Map-Reduce Workflow Scheduling Framework over Hadoop Clusters

Optimizing Internal Overlaps by Self-Adjusting Resource Allocation in Multi-Stage Computing Systems

Task Scheduling for Heterogeneous Computing Based on Bayesian Optimization Algorithm

A Hybrid Ant Colony Optimisation Algorithm for Job Shop Problems and Its Convergence Analysis.

Resource Constrained Job Scheduling with Parallel Constraint-Based ACO

Job Scheduling in Cloud Computing Using a Modified Harris Hawks Optimization and Simulated Annealing Algorithm

Task Scheduling Using Bayesian Optimization Algorithm for Heterogeneous Computing Environments

A Research on Genetic Algorithm-Based Task Scheduling in Cloud-Fog Computing Systems

Hadoop Scheduling Base On Data Locality

MQWAGS: Research on Job Scheduling Algorithms Based on Cloud Computing