Abstract:SummaryLong‐running stream applications usually share the same fundamental computational infrastructure. To improve the efficiency of data processing in stream processing systems, a data analysis operator could be partitioned into n parallel tasks. The partitioned tasks are usually deployed on m nodes coexisting with other application operators. Because the node performance can vary in unpredictable ways (i.e., (1) stream input rates may fluctuate and (2) computational resource availability varies as other applications are affected), the nodes have different processing steps, and the slow node determines the operator performance. Hence, the tasks should be redistributed at runtime for stream applications to meet their strict latency requirements. Our key idea is to redistribute the tasks to the best node dynamically adaptive to resource or load fluctuations. In this paper, we present a runtime‐aware adaptive schedule mechanism that aims at minimizing the operator processing latency and minimizing the latency difference between different nodes' tasks. We propose a new abstraction called performance cost ratio (PCR) that evaluates the node performance. The higher the node's PCR is, the less cost the node will pay for processing one tuple, and the more tasks should be deployed on it. In a scheduling, we first sort tasks descendingly by their loads and sort nodes by their PCR. Then we reassign the amount of computation according to the node's PCR to keep the node's PCR and its input rate the same or in similar proportion in all PCRs. The PCR‐based quantitative algorithm applies itself to make tasks loads quantized to the processing capacity of nodes, move the minimum amount of operator's tasks, and keep the tasks local at the same time. We have implemented a runtime‐aware adaptive scheduler as an extension to Storm and evaluated this strategy. We achieve the optimization goal using less computational resources. Copyright © 2015 John Wiley & Sons, Ltd.

Supporting Self-Adaptation in Streaming Data Mining Applications

Supporting a Visualization Application on a Self-Adapting Grid Middleware

Self-adaptive Change Detection in Streaming Data with Non-Stationary Distribution

Resource Allocation in a Middleware for Streaming Data

Adaptive Load Management over Real-Time Data Streams

A Static Resource Allocation Framework for Grid-based Streaming Applications.

Supporting a real-time distributed intrusion detection application on GATES

Gates: A Grid-Based Middleware for Processing Distributed Data Streams

A Control-based Approach Towards Adaptive Stream Processing Student :

Variance Reduced Diffusion Adaptation for Online Learning over Networks

On Appropriate Assumptions to Mine Data Streams: Analysis and Practice

Maximum Sustainable Throughput Evaluation Using an Adaptive Method for Stream Processing Platforms.

Adaptive Scheduling for Efficient Execution of Dynamic Stream Workflows

Building User-defined Runtime Adaptation Routines for Stream Processing Applications

Supporting Fault-Tolerance in Streaming Grid Applications

Concurrent and Storage-Aware Data Streaming for Data Processing Workflows in Grid Environments

Adaptive Data Analysis for Growing Data

Achieving Self-Aware Parallelism in Stream Programs

Towards Smooth and High-Quality Bitrate Adaptation for HTTP Adaptive Streaming

Runtime-Aware Adaptive Scheduling in Stream Processing

An adaptive algorithm for dealing with data stream evolution and singularity