On-demand File-Parallel Transfer Algorithm Supporting Virtual Cluster Migration

Recently, the virtualization technologies are widely integrated with grid technology.Virtual cluster in distributed environment is commonly used to solve a variety of parallelprocessing problems. The dynamic nature of the virtual cluster and virtual cluster migration,making the virtual cluster applications often involve the transfer of large files to the physicalcluster. With the popularity of computer and Internet technology, massive amount of data isproduced every day, and has a speed of the explosive growth. Grid computing technology alsomakes large-scale data-intensive applications widely develop. In such applications, the datageneration clusters, the data storage clusters and data processing clusters of large amount ofdata are often distributed in the physical location of the distant. These data needs to aggregatefrom local computers to remote data processing center for computing, displaying or storing.So, it often requires the efficient transfer of large amounts of data over wide-area distributedcomputing environments. In parallel processing, we want that the large data files aretransmitted to the processing nodes of each other within the shortest possible time, so that thedata processing can be performed simultaneously. The rapid transfer of large files among multipleclusters gradually becomes a hot research.At present, there are many researches about file transfer algorithms over WAN and theapplication technologies to improve file transfer performance. At this stage, the research offile transfers is focused on increasing the degree of parallelism of the transmission, improvingnetwork bandwidth utilization, and shortening the overall transmission time of the batch filerequests. In some work, it improves the file transfer performance by using multi-hop pathsplitting and multi-pathing transmission and source-side parallel to implement of the paralleltransmission of the file. But it did not take into account that files are received parallelly inmultiple destinations which is better to improve the degree of parallelism. The currentresearches also did not take into account the global control of batch file requests across thenetwork.This paper presents an On-demand File-Parallel Transfer (OFPT) algorithm, which aimsat minimizing the makespan of overall file transfer requests. Taking advantage of the high transfer performance within a single cluster, the algorithm performs simultaneous transfer ofdifferent pieces of files to multiple nodes of a cluster when one node of the cluster requeststhe files. It employs multi-hop path splitting methods to achieve parallel transmission. Foreach path of a single source-destination pair, OFPT employs multi-pathing methods and adjustthe limit of path’s hop flexibility to improve the transfer performance. When several requestscompete for the physical links, the algorithm evenly allocate the bandwidth resources uponthe transmission load of each request, to make full use of the network bandwidth, to transferthe bulk transfer request rapidly.We carried out extensive simulation experiments using NS2. The results show that ourOFPT algorithm make effective use of the network resources and presents better transferperformance compared with the current large file transfer algorithms in throughout andnetwork bandwidth utilization, attained the expected results of our work.