Symbiosis: Network-aware Task Scheduling in Data-Parallel Frameworks

Jingjie Jiang,Shiyao Ma,Bo Li,Baochun Li

DOI: https://doi.org/10.1109/infocom.2016.7524415

2016-01-01

Abstract:Even with the recent proliferation of in-memory computation in data-parallel frameworks (such as Spark), transfers over the network are still time-consuming. Similar to computation, network transfers serve as main roadblocks as we try to minimize job completion times. Existing schedulers were designed as isolated solutions that focused on computation or network performance only. Without any coordination, the utilization of computation and network resources may become unbalanced, leading to a reduced level of overall resource utilization. In this paper, we design, implement, and evaluate Symbiosis, a network-aware task scheduler designed to coordinate computation-bound and network-bound tasks in a large cluster, so that resources are utilized in a more balanced fashion. Symbiosis is an online scheduler that predicts resource imbalance before launching tasks, and correct such imbalance by co-locating computation-bound and network-bound tasks in the same executor process. As a guiding principle, it is engineered to be practically implemented within and to complement existing data-parallel frameworks. We have implemented Symbiosis within Spark, and carried out our experiments on a 100-node cluster. We show convincing evidence that Symbiosis reduces job completion times by 11.9% in comparison to Spark's current scheduler with little overhead.