Communication-Efficient Task Scheduling for Real-Time Distributed Computing.

Jiannan Pang,Wei Wang,Zhaoyang Zhang

DOI: https://doi.org/10.1109/wcsp52459.2021.9613424

2021-01-01

Abstract:In the distributed big data real-time computing platform, the task scheduling has significant impact on the real-time performance. In this paper, we first reveal the inefficiency of the conventional real-time streaming processing scheduling algorithm in the well-known Storm framework. Based on our observations, we propose a communication-efficient task scheduling algorithm taking both physical cluster's computation and communication capacities into consideration. Our goal is to minimize the communication cost under taking full advantage of cluster computing capacity. Due to the non-convex constraints in our problem, we relax the binary integer to a continuous interval so that the condition can meet to handle the $l_{0}$-norm constraint. By introducing an auxiliary variable and constructing penalty function, $l_{0}$-norm can be transformed to obtain a solution for the relaxed problem. In view of the bipartite graph property of our model, we round the fractional solution of the relaxed problem to the integer solution in its sub-graphs to approach the solution of the original problem. Finally, to evaluate the proposed method, we compare it with two conventional scheduling algorithms, which demonstrates that the proposed algorithm outperforms the other two baseline schemes.