Topology-Aware Task Allocation For Distributed Stream Processing With Latency Guarantee

Xiaohui Wei,Xun Wei,Hongliang Li,Yuan Zhuang,Hengshan Yue

DOI: https://doi.org/10.1145/3239576.3239621

2018-01-01

Abstract:Nowadays it becomes more and more critical to process the increasingly large amounts of data in timely manner. In order to meet this requirement and ensure the reliable processing of streaming data, a variety of distributed stream processing architectures and platforms have been developed, which handles the fundamental task of allocating processing tasks to the currently available physical resources and routing streaming data between these resources. However, many stream processing systems lack an intelligent scheduling mechanism, in which their default schedulers allocate tasks without taking resource demands and availability, or the transfer latency between resources into consideration. Besides stream processing has a strict request for latency. Thus it's important to give latency guarantee for distributed stream processing. In this paper, we propose a new algorithm for stream processing with latency guarantee, the algorithm both consider transfer latency and resource demand in the process of task allocation. Extensive experiments verify the correctness and effectiveness of our approach. Under the condition of satisfying the latency constraints, the heuristic algorithm AHA on average, reduce more than 21.3% and 58.9% resources compared with the greedy and the round-robin algorithms.