Research On Load Balancing Strategy Of Non-metadata Distributed Storage System

In today's increasingly large scale of data,the storage capacity of a single machine is obviously no longer able to meet the data storage needs of enterprises.Distributed storage systems have emerged in this context.The scale of nodes in a distributed storage system is huge,and the configuration of each node is different.Different types of requests bring different load pressures on the nodes,which brings huge challenges to load management.The load balancing algorithm of a non-metadata storage system needs to ensure that massive amounts of data can be distributed to many storage nodes of the system according to the load conditions of different nodes.In the case of uneven load in the system,load data migration is used to balance the load between nodes.At the same time,it also caused the degradation of the system's service capabilities.In view of the degradation of service capabilities of non-metadata distributed storage systems due to load balancing,the paper proposes a load timing prediction strategy based on the ARIMA model,according to the CPU,memory,disk and network of the nonmetadata distributed storage system.Historical load timing information such as bandwidth,predict the load change trend of the cluster in a period of time in the future,and better allocate resources based on the prediction result;propose a load balancing strategy based on dynamic thresholds,use load forecast change trends and system real-time load status information,Calculate the threshold of the load balancing strategy suitable for the current state of the system,determine the minimum amount of data that needs to be migrated to balance the load between nodes,solve the resource demand contradiction between system load balancing and user I/O,and sacrifice part of the load balance when the system is under high load Balance can significantly improve the efficiency of user I/O,use the idle resources of nodes to complete load migration when the system is under low load,and reduce the impact of load migration between nodes on the quality of system service under high load conditions.The experimental results show that the Swift system adopts a dynamic threshold-based load balancing strategy,which has achieved significant improvements in reducing system load imbalance,improving system resource utilization,and improving user I/O efficiency.Compared with the traditional Swift load balancing algorithm,the variance of load fluctuations at different moments has been reduced by more than 30%,the load balance between nodes has been improved by more than 50%,and the completion time of user I/O requests has been reduced by more than 10%;Compare with load balancing algorithm based on the I/O,the node's I/O completion time has been shortened by 7%~8% on average,and the load volatility has been reduced by more than 40%.