Abstract:Traditional High-Performance Computing (HPC) based big-data applications are usually constrained by having to move large amount of data to compute facilities for real-time processing purpose. Modern HPC systems, represented by High-Throughput Computing (HTC) and Many-Task Computing (MTC) platforms, on the other hand, intend to achieve the long-held dream of moving compute to data instead. This kind of data-aware scheduling, typically represented by Hadoop MapReduce, has been successfully implemented in its Map Phase, whereby each Map Task is sent out to the compute node where the corresponding input data chunk is located. However, Hadoop MapReduce limits itself to a one-map-to-one-reduce framework, leading to difficulties for handling complex logics, such as pipelines or workflows. Meanwhile, it lacks built-in support and optimization when the input datasets are shared among multiple applications and/or jobs. The performance can be improved significantly when the knowledge of the shared and frequently accessed data is taken into scheduling decisions.To enhance the capability of managing workflow in modern HPC system, this paper presents CloudFlow, a Hadoop MapReduce based programming model for cloud workflow applications. CloudFlow is built on top of MapReduce, which is proposed not only being data aware, but also shared-data aware. It identifies the most frequently shared data, from both task-level and job-level, replicates them to each compute node for data locality purposes. It also supports user-defined multiple Map- and Reduce functions, allowing users to orchestrate the required data-flow logic. Mathematically, we prove the correctness of the whole scheduling framework by performing theoretical analysis. Further more, experimental evaluation also shows that the execution runtime speedup exceeds 4X compared to traditional MapReduce implementation with a manageable time overhead. (C) 2014 Elsevier B.V. All rights reserved.

State-of-art Research of Cluster Resource Management in Dataflow Computing Model

Performance-Driven Task and Data Co-scheduling Algorithms for Data-Intensive Applications in Grid Computing

A Novel Job Scheduling Model to Enhance Efficiency and Overall User Fairness of Cloud Computing Environment.

A Deadline-Aware Coflow Scheduling Approach for Big Data Applications.

On Global Resource Allocation in Clusters for Data Analytics.

Cloudflow: A Data-Aware Programming Model for Cloud Workflow Applications on Modern Hpc Systems

Research on the Framework and Resource Scheduling Mechanisms of Hadoop YARN

THE RESEARCH ON RESOURCE ALLOCATION IN COMPUTATIONAL GRID

Coflow Scheduling in the Multi-Resource Environment

Adaptive priority-based data placement and multi-task scheduling in geo-distributed cloud systems

A Policy of Task Allocation Base on Distributed Cluster Computing Towards Cloud

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Distributed Bottleneck-Aware Coflow Scheduling in Data Centers

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

Data Diffusion: Dynamic Resource Provision and Data-Aware Scheduling for Data Intensive Applications

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Clustering-based Multi-objective Optimization Considering Fairness for Multi-workflow Scheduling on Clouds

Multi-policy-aware MapReduce Resource Allocation and Scheduling for Smart Computing Cluster.

Heterogeneous Cluster Application Communication Optimization and Computer Big Data Management

Optimization of Big Data Parallel Scheduling Based on Dynamic Clustering Scheduling Algorithm

Multiple Resources Scheduling for Diverse Workloads in Heterogeneous Datacenter