The End-to-End Transfer Strategy Of Lots Of Small Files For GridFTP

The transmission time of small file is concentrated in the slow start phase,and there is an idle time between two different file transmissions.Too much time wasted resulting in low transmission throughput when transferring a large number of small files.GridFTP is an improved protocol based on FTP that proposes pipelining parameter to improve the performance of a large number of small file transmission,but the pipelining parameter is set statically,resulting in poor improvement.Aiming at the lots of small files(LOSF)transmission problems,a flexible storage structure is proposed to reclassify files,and the transmission queue is rearranged for different types of file sets to optimize transmission throughput with this structure.There are three optimization method for different file sets.Firstly,The BD algorithm is mainly optimized for the transmission of ordinary file sets.The optimal pipelining value is determined by the situation of the files are classified,and the file transmission queue is constructed according to the parameter value.Secondly,The BDH algorithm optimizes the transmission of file sets containing high-priority files.The purpose is to ensure that high-priority files can be delivered as soon as possible while maintaining the optimized throughput.Thirdly,The BDR algorithm optimizes the transmission of the file set with strict transmission order,and allows the files to reach the destination continuously while ensuring the transmission throughput.The experiment includes different types of file sets,different network bandwidths and different network delay conditions.The experimental results show that the throughput of BD only decrease 8.9% when double the RTT.When a small number of large files are added into the file set,the throughput of the BD increases about 15%.When the file set containing special files,the throughput fluctuation is small.