DDLB: A Dynamic and Distributed Load Balancing Strategy.
Bingqi Liu,Jiulong Chang,Limin Xiao,Guangjun Qin,Bing Wei,Zhisheng Huo
DOI: https://doi.org/10.1109/hpcc/smartcity/dss.2019.00266
2019-01-01
Abstract:In view of the problem of load imbalance for distributed file systems, the current solutions either adopt centralized algorithm and implement load balancing by increasing the number of files as the replica, or lack considerations for network transmission and network bandwidth. Therefore, the current solutions are relatively inefficient in distributed file system. To solve this problem, this paper proposes DDLB, a dynamic and distributed load balancing algorithm which is completely based on distributed load balancing architecture. DDLB can not only monitor the IO load on the data server dynamically and real time, but also use the load prediction model, which based on weighted mean, to reduce the impact of network delay and collection delay on load decision-making. In order to minimize the frequency of remote load collection and decrease network bandwidth occupation, DDLB adopts the remote distributed load collection method based on threshold and the cooling collection mechanism. Moreover, dynamic replica management mechanism of DDLB takes data blocks instead of files as the basic unit for replica replication, and reduces the IO load of the data server by adding replicas of hotspot data blocks. For proving the effectiveness of DDLB, this paper conducts a comprehensive evaluation in the HDFS cluster environment. The results show that compared with the traditional load balancing algorithm, DDLB can effectively balance the IO load between data servers in the distributed file system and improve the data access performance of IO-intensive applications.