Design And Implementation Of Data Transfer In Global Distributed Storage System

Data-intensive and high-performance computing applications are beyond thecapability of the traditional data management system. Luckily, the storagevirtualization technique provides an effective way to solve it. GDSS, which makes the best use of the storage virtualization technique byrealizing the unified management of the storage resources distributed in wide area,provides a logical view and reduces the management cost of the system. The self-adapt transfer system is an important module in GDSS, which infectsthe performance of GDSS greatly. In order to guarantee the high-performance andhigh-availability, it adopts several techniques, such as self-adapt transfer, separate andparallel transfer, connection reuse, breakpoint resume, and so on. Self-adapt transfertechnique decides the transfer mode by the size of the files and the status of theservers. Separate and parallel transfer technique is used in transferring big files, whileconnection reuse technique, which transfers batches of small files by only one dataconnection to reduce the connection times, is used when transferring batches of smallfiles. On the other hand, in order to guarantee the availability of the system,breakpoint resume technique that takes advantage of the available repetitions in therecovery process and double log technique that uses both main log and backup log tomake sure that at least one log is available at any time are adopted in GDSS. Test shows that the performance of GDSS increases 1 to 3 times than that of thegeneral FTP when transferring big files, and 2 to 10 times when transferring batchesof small files. It proves that the transfer system of GDSS has met the requirements.