Hypergraph-partitioning-based online joint scheduling of tasks and data
Yao Song,Liang Wang,Limin Xiao,Wei Wei,Rafał Scherer,Guangjun Qin,Jinquan Wang
DOI: https://doi.org/10.1007/s11227-022-04460-0
IF: 3.3
2022-04-30
The Journal of Supercomputing
Abstract:Recently, wide-area distributed computing environments have become popular owing to their huge resource capability. In a wide-area distributed computing environment, joint scheduling of tasks and data is the main strategy to improve system performance. However, the geographically distributed diverse resources exhibit high variations, making it challenging to design efficient joint scheduling of tasks and data. To accurately adapt to the dynamic variations of geographically distributed diverse resources and achieve a high system performance, this study proposes a hypergraph-partitioning-based online joint scheduling method. The proposed method constructs a hypergraph of geographically distributed tasks, data, and diverse resources to clearly describe the correlation among the three elements and quantitatively reflect the time cost of different process in the environment. The hypergraph is dynamically updated according to the generated scheduling scheme and the collected information to reflect the dynamic variations of resource states. Then, a hypergraph partition optimization mechanism is proposed to generate efficient joint scheduling schemes, thus reducing the overall completion time in the system. The experimental results indicate that compared with the state-of-the-art joint scheduling methods, the proposed method reduces the overall completion time by up to 25.67% and significantly reduces the task waiting time, although it makes a concession in the data migration time.
computer science, theory & methods,engineering, electrical & electronic, hardware & architecture