Abstract:An important challenge for the adoption of cloud computing in the scientific community remains the efficient allocation and execution of data-intensive scientific workflows to reduce execution time and the size of transferred data. The transferred data overhead is becoming significant with emerging scientific workflows that have input/output files and intermediate data products ranging in the hundreds of gigabytes. The allocation of scientific workflows on public clouds can be described through a variety of perspectives and parameters, and has been proved to be NP-complete. This paper proposes an evolutionary approach for task allocation on public clouds considering data transfer and execution time. In our framework, a solution is represented using an allocation chromosome that encodes the allocation of tasks to nodes, and an ordering chromosome that defines the execution order according to the scientific workflow representation. We propose a multi-objective optimization that relies on a cloud cost model and employs tailored evolution operators. Starting from a population of possible solutions, we employ crossover and mutation operators on both chromosomes aiming at optimizing the data transferred between nodes as well as the total workflow runtime. The crossover operators combine parts of solutions to reduce data overhead, whereas the mutation operators swamp between parts of the same chromosome according to pre-defined rules. Our experimental study compares between the proposed approach and current state-of-the art approaches using synthetic and real-life workflows. Our algorithm performs similarly to existing heuristics for small workflows and shows up to 80 % improvements for larger synthetic workflows. To further validate our approach we compare between the allocation and scheduling obtained by our approach with that obtained by popular scientific workflow managers, when real workflows with hundreds of tasks are executed on a public cloud. The results show a 10 % improvement in runtime over existing schedulers, caused by a 80 % reduction in transferred data and optimized allocation and ordering of tasks. This improved data locality has greater impact as it can be employed to improve and study data provenance and facilitate data persistence for scientific workflows.

On Scheduling of High-Throughput Scientific Workflows under Budget Constraints in Multi-Cloud Environments.

Cost-Efficient Workflow Scheduling Algorithm for Applications With Deadline Constraint on Heterogeneous Clouds

Online Optimization Scheduling for Scientific Workflows with Deadline Constraint on Hybrid Clouds

A Budget and Deadline Aware Scientific Workflow Resource Provisioning and Scheduling Mechanism for Cloud

Throughput-Guarantee Resource Provisioning for Streaming Analytical Workflows in the Cloud

Budget-deadline Constrained Approach for Scientific Workflows Scheduling in a Cloud Environment

Cost-Driven Scheduling for Deadline-Based Workflow Across Multiple Clouds.

Cost and makespan-aware workflow scheduling in hybrid clouds.

Cost-Driven Scheduling For Deadline-Constrained Workflow On Multi-Clouds

Scaling and Scheduling to Maximize Application Performance within Budget Constraints in Cloud Workflows

Science in the Cloud: Allocation and Execution of Data-Intensive Scientific Workflows

Cost-Efficient Scheduling of Workflow Applications with Deadline Constraint on IaaS Clouds

A Deadline-Constrained Scheduling Algorithm for Scientific Workflows in Clouds

End-to-End Delay Minimization for Scientific Workflows in Clouds under Budget Constraint

Workflow Scheduling in the Cloud with Weighted Upward-rank Priority Scheme Using Random Walk and Uniform Spare Budget Splitting

Cost-Driven Scheduling Strategy for Scientific Workflow under Multi-cloud Environment

Cost Optimization for Scheduling Scientific Workflows on Clouds under Deadline Constraints.

A Data and Task Co-Scheduling Algorithm for Scientific Cloud Workflows

A Security and Cost Aware Scheduling Algorithm for Heterogeneous Tasks of Scientific Workflow in Clouds

An Optimizing Algorithm for Deadline Constrained Scheduling of Scientific Workflows in IaaS Clouds Using Spot Instances.

A Multi-stage Dynamic Game-Theoretic Approach for Multi-Workflow Scheduling on Heterogeneous Virtual Machines from Multiple Infrastructure-as-a-Service Clouds.