Abstract:Recently, cloud computing has emerged as a promising computing infrastructure for performing scientific workflows by providing on-demand resources. Meanwhile, it is convenient for scientific collaboration since different cloud environments used by the researchers are connected through Internet. However, the significant latency arising from frequent access to large datasets and the corresponding data movements across geo-distributed data centers has been an obstacle to hinder the efficient execution of data-intensive scientific workflows. In this paper, we propose a novel graph-cut based data and task co scheduling strategy for minimizing the data transfer across geo-distributed data centers. Specifically, a dependency graph is firstly constructed from workflow provenance and cut into sub graphs according to the datasets which must appear in fixed data centers by a multiway cut algorithm. Then, the sub graphs might be recursively cut into smaller ones by a minimum cut algorithm referring to data correlation rules until all of them can well fit the capacity constraints of the data centers where the fixed location datasets reside. In this way, the datasets and tasks are distributed into target data centers while the total amount of data transfer between them is minimized. Additionally, a runtime scheduling algorithm is exploited to dynamically adjust the data placement during execution to prevent the data centers from overloading. Simulation results demonstrate that the total volume of data transfer across different data centers can be significantly reduced and the cost of performing scientific workflows on the clouds will be accordingly saved.

Scientific Workflow Scheduling Based on Data Transformation Graph for Remote Sensing Application

Workflow Scheduling for Remote Sense Application Using Data Transformation Graph in Inter-cloud

Exploiting Scalable Parallelism for Remote Sensing Analysis Models by Data Transformation Graph.

Big Data Processing Workflows Oriented Real-Time Scheduling Algorithm using Task-Duplication in Geo-Distributed Clouds

A Cloud Computing Workflow Scheduling Method Based on DTGA

Cost-Efficient Workflow Scheduling Algorithm for Applications With Deadline Constraint on Heterogeneous Clouds

Graph partition–based data and task co‐scheduling of scientific workflow in geo‐distributed datacenters

Genetic Programming for Dynamic Workflow Scheduling in Fog Computing

Fault-tolerant scheduling and data placement for scientific workflow processing in geo-distributed clouds

Efficient Data and Task Co-Scheduling for Scientific Workflow in Geo-Distributed Datacenters

A Data and Task Co-Scheduling Algorithm for Scientific Cloud Workflows

Cost-Efficient Task Scheduling for Geo-distributed Data Analytics.

Improved Hungarian algorithm–based task scheduling optimization strategy for remote sensing big data processing

An Efficient and Scalable Framework for Processing Remotely Sensed Big Data in Cloud Computing Environments

Graph-Cut Based Coscheduling Strategy Towards Efficient Execution of Scientific Workflows in Collaborative Cloud Environments

A Clustering Based Coscheduling Strategy for Efficient Scientific Workflow Execution in Cloud Computing.

Accelerating Remote Sensing Data Analysis Workflows by On-demand Processing

Dynamic Neighborhood Grouping-Based Multi-Objective Scheduling Algorithm for Workflow in Hybrid Cloud

A Study of Grid Workflow Dynamic Customization for Remote Sensing Quantitative Retrieval

Dynamic Fault-Tolerant Workflow Scheduling with Hybrid Spatial-Temporal Re-Execution in Clouds

A Two-Step Data Placement and Task Scheduling Strategy for Optimizing Scientific Workflow Performance on Cloud Computing Platform