On-demand File Transfer Algorithm Supporting Data-parallel over Multi-cluster

WEI Xiao-hui,ZHOU Qin-qin,LI Hong-liang
DOI: https://doi.org/10.3969/j.issn.1002-137x.2013.03.016
2013-01-01
Computer Science
Abstract:Data-intensive applications often require the efficient transfer of large amounts of data over wide-area distribu-ted computing environments.In parallel processing,such data transmissions usually happen between data generation clusters and the data storage or processing clusters from time to time.This paper presented an on-demand file transfer algorithm,which aims at minimizing the makespan of overall file transfer requests.Taking advantage of the high transfer performance within a single cluster,the algorithm performs simultaneous transfer of different pieces of files to multiple nodes of a cluster when one node of the cluster requests the files.It employs multi-hop path splitting and multi-pathing methods in [1] to improve the transfer performance of a single source-destination pair.When several requests compete for the physical links,the algorithm evenly allocates the bandwidth resources upon the transmission load of each request,to make full use of the network bandwidth.
What problem does this paper attempt to address?