Abstract:Performance of data streaming applications is co-determined by both networking and computing resources, and therefore they should be co-scheduled and co-allocated in an integrated and coordinated way. Dynamic control of resource scheduling and allocation is required, because unilateral redundancy in either networking or computing resources may result in the overprovision of it and the other may become a bottleneck. To avoid resource shortage as well as overprovision, in this paper, a virtualized platform is utilized to implement data streaming and processing. In this platform, fuzzy logic controllers are designed to allocate CPU resources; iterative bandwidth allocation is applied and is processing-and storage-aware to guarantee on-demand data provisioning. Experimental results show that our approach leads to higher application performance as well as higher resource utilization, compared with other resource scheduling and allocation methods.Note to Practitioners-Data streaming has become an important paradigm in many business and scientific applications, such as fraud detection in banking industry and gravitational-wave observation in astronomy. In this paper, we apply virtualization to provide better support for these applications, leveraging its advantage in resource on-demand allocation. A novel dynamic control method is proposed so that CPU and bandwidth can be co-scheduled and co-allocated since for data streaming applications these resources are tightly coupled from the performance perspective. In this control method, fuzzy logic control is applied for CPU allocation and an iterative algorithm is adopted for processing-, congestion-, and storage-aware bandwidth allocation. We use fuzzy control because it does not rely on the mathematical modeling of an object and can implement human experts' heuristic knowledge via IF-THEN rules. Therefore, it is desirable in our data streaming scenario, due to the variable coupling and heavily nonlinear nature of the system.

Concurrent and Storage-Aware Data Streaming for Data Processing Workflows in Grid Environments

Storage Aware Resource Allocation for Grid Data Streaming Pipelines

Block-Based Concurrent and Storage-Aware Data Streaming for Grid Applications with Lots of Small Files

Performance-Driven Task and Data Co-scheduling Algorithms for Data-Intensive Applications in Grid Computing

Scalable-Grain Pipeline Parallelization Method For Multi-Core Systems

An Integrated Resource Management and Scheduling System for Grid Data Streaming Applications

Agile Data Streaming for Grid Applications

Scheduling On-demand Data Streaming for Cyberinfrastructure Applications with Constraints of Storage and Bandwidth

Streaming Task Graph Scheduling for Dataflow Architectures

Progressive online aggregation in a distributed stream system

A Time-driven Data Placement Strategy for a Scientific Workflow Combining Edge Computing and Cloud Computing

An Efficient Scheduling Algorithm for Stream Computing.

Energy-aware application scheduling based on genetic algorithm.

Genetic Programming for Dynamic Workflow Scheduling in Fog Computing

GA-based QOS-aware workflow scheduling of deadline tasks in grid computing

Big Data Processing Workflows Oriented Real-Time Scheduling Algorithm using Task-Duplication in Geo-Distributed Clouds

Gridflow: Workflow Management for Grid Computing

Dynamic Control of Data Streaming and Processing in a Virtualized Environment

A Data-Driven Architecture Design of Stream Computing for the Dispatch and Control System of the Power Grid

A Grid Scheduling Approach Based on Stream Jobs

Grid workflow scheduling based on task and data locations