Hadoop Data Load Balancing Method Based on Dynamic Bandwidth Allocation

Wei-wei Lin,Bo Liu
DOI: https://doi.org/10.3969/j.issn.1000-565X.2012.09.008
2012-01-01
Abstract:Data load balancing greatly affects the performance of the Hadoop distributed file system (HDFS). In order to overcome the inefficiency and inflexibility of the default data load balancing method in HDFS, this paper devises a novel dynamic load balancing method, which dynamically allocates network bandwidth to achieve the data load balancing by controlling variables. Then, the corresponding mathematical model is constructed based on the controlled variables. Experimental results show that the devised method can not only guarantee the performance of the HDFS data access system but also improve the data load balancing efficiency in the presence of a new cluster node.
What problem does this paper attempt to address?