The Design And Implementation Of Load Balancing In The PNFS

Load balancing technology is a common and indispensable mean for optimization inthe parallel network file systems. With the help of the load analysis and some loadbalancing algorithms, it can be effective to make the load evenly distributed to differentstorage nodes, which in order to improve overall system stability and quality of service.In parallel file systems, with the storage node add, delete, file creation, deletion andmodification, and the explosive growth of the number of visits, the load of each storagenode changes dynamically in unpredictable ways, which often faces with excessive loss ofsome server resources, but low resource utilization of other servers, resulting in tilting ofthe load. Meanwhile node failures often cause the system not working properly. Althoughreplica strategy can help to solve this problem, in different scenarios, it is needed for thevarious features of the scene, based on replica strategies and around the load balancingtechnology research to identify appropriate strategies and timing. In this paper, a dynamicload balancing mechanism based on probability distribution PDDB(ProbabilityDistribution Dynamic Balance) is designed for the typical parallel network file system----pNFS.In parallel network filesystem, PDDB uses file replica strategy and placing filereplicas according to capacity balance, with mirroring and interleaving placement mode.The data servers are built on adaptive load monitor system. And the metadata servercollects the information of storage nodes as the useage of CPU, memory, storage space,network bandwidth and so on. After information integration, PDDB uses the method of theprobability distribution, and obtains an integrated load and inference load trends withhistorical data. Then it could migrate hot file between each nodes and avoid agglomeration.At the same time the metadata management of replicas and replica consistencymaintenance, ensure that when a copy of any storage node fails, you can get file contentsthrough other normal replica to guarantee the normal operation of the system.In the overall testing, compared with randomized balance system and least load firstbalance system, our system makes intelligent balance in the load gap between the storagenodes, reduced by42%and30%respectively. In the case of the same total load, the loadvariation of each node is relatively flat, and the average load reduces at least10%. Theaverage network throughput of the file system increases by20%, in addition the reliability,stability and scalability are improved.