Abstract:SummaryLong‐running stream applications usually share the same fundamental computational infrastructure. To improve the efficiency of data processing in stream processing systems, a data analysis operator could be partitioned into n parallel tasks. The partitioned tasks are usually deployed on m nodes coexisting with other application operators. Because the node performance can vary in unpredictable ways (i.e., (1) stream input rates may fluctuate and (2) computational resource availability varies as other applications are affected), the nodes have different processing steps, and the slow node determines the operator performance. Hence, the tasks should be redistributed at runtime for stream applications to meet their strict latency requirements. Our key idea is to redistribute the tasks to the best node dynamically adaptive to resource or load fluctuations. In this paper, we present a runtime‐aware adaptive schedule mechanism that aims at minimizing the operator processing latency and minimizing the latency difference between different nodes' tasks. We propose a new abstraction called performance cost ratio (PCR) that evaluates the node performance. The higher the node's PCR is, the less cost the node will pay for processing one tuple, and the more tasks should be deployed on it. In a scheduling, we first sort tasks descendingly by their loads and sort nodes by their PCR. Then we reassign the amount of computation according to the node's PCR to keep the node's PCR and its input rate the same or in similar proportion in all PCRs. The PCR‐based quantitative algorithm applies itself to make tasks loads quantized to the processing capacity of nodes, move the minimum amount of operator's tasks, and keep the tasks local at the same time. We have implemented a runtime‐aware adaptive scheduler as an extension to Storm and evaluated this strategy. We achieve the optimization goal using less computational resources. Copyright © 2015 John Wiley & Sons, Ltd.

Adaptive Scheduling Strategy For Data Stream Management System

An On-the-Fly Scheduling Strategy for Distributed Stream Processing Platform.

An Efficient Scheduling Algorithm for Stream Computing.

Adaptive Scheduling for Shared Window Joins over Data Streams.

Towards Scheduling to Maximize Weighted Delivered Data in Maritime CPSs

The Research Of Real-Time Scheduling For Data Stream Operators

An effective scheduling algorithm based-on QoS adaptation framework over data streams

DRS: Dynamic Resource Scheduling for Real-Time Analytics over Fast Streams

ADSM: Adaptive Data Scheduling Method for Hybrid Memories in Distributed System

A state lossless scheduling strategy in distributed stream computing systems

Low Latency and High-Reliability Resource Scheduling Method in Big Data Streaming Computing Environment

Fair and Delay Adaptive Scheduler (FDAS) Preliminary Modeling and Optimization

Adaptive Scheduling Parallel Jobs with Dynamic Batching in Spark Streaming

An adaptive strategy for scheduling data-intensive applications in Grid environments

Runtime-Aware Adaptive Scheduling in Stream Processing

A Deadline-Aware Coflow Scheduling Approach for Big Data Applications.

Adaptive Scheduling Framework of Streaming Applications based on Resource Demand Prediction with Hybrid Algorithms

Adaptive Scheduling for Efficient Execution of Dynamic Stream Workflows

Adaptive Query Scheduling in Key-Value Data Stores.

Automating Characterization Deployment in Distributed Data Stream Management Systems.

Multi-aggregate-query Scheduling over Data Streams