Abstract:Load balancing is the salient problem in data stream processing systems and also in complex event processing systems. And the imbalance in operational workers becomes extremely apparent as the scale is up with more workers and skewed datasets. In this paper, we find out that the upstream skewed sources can also exacerbate the load imbalance in the downstream workers and this bottleneck cannot be handled well by existing schemes. Thus, we propose a novel stream partitioning solution called BACK PROPAGATION GROUPING (BPG), and its core components are key splitting, back propagation and calibration signal. We verify BPG in theory and test it on both real-world and synthetic data streams. The results show that the imbalance is 10–100x less with BPG than with the previous state-of-the-art. And this metric translates into an improvement of up to 34% in throughput when deployed on Apache Storm cluster. In conclusion, BPG mitigates the load imbalance dramatically on highly skewed datasets and especially when the sources are also skewed.

Back Propagation Grouping: Load Balancing at Global Scale When Sources Are Skewed