The Research Of Load Rebalancing Scheme In Distributed File System

Distributed File System gradually develops to Cloud Storage from initial Network File System. Cloud Storage is a new formed and extended concept of Cloud Computing. In the large cluster and dynamic cloud storage system, the central node will become the performance bottleneck of the entire system, as the increased number of files uploading and file access. Therefore, it has been an important issue to reduce the dependency of the central node in a distributed file system. Recently, it is concerned on the distributed file system based on DHT because of the feature of non-center node. But it cannot keep balancing as existing of the hot node and file modified. Load Balancing is also an important issue. The status of cluster affects network throughput. It can avoid hot spots and improve the response speed. This paper will study the issues of load rebalancing in distributed file system based on DHT deeply.Through researching existing Load Rebalancing algorithm, we found that it will lead to the phenomenon of jitter in the data migration process due to partial node information, and external high load node. It will result in load balancing efficiency. Therefore, we propose two major aspects. Firstly, it expands the number of the random selected node sample so that the estimated values of node load close to the real value. Secondly, we choose the successor node by a two-way extends to find out right node, thereby reducing redundant data migration operations.Replica management and node selection are closely related to the distributed file system. Thus, in the actual production environment, it’s different between the strategy of replica assignment and the strategy of file uploading base on DHT. Therefore, we use the soft link to redirect the location of replica. Meanwhile, we propose node selection strategy to avoid conflict when multiple light load nodes simultaneously choose the same high load node during the process of data migration. We map node logic location to physical location by Space-filling Curve, and convert the heterogeneous cluster into the homogeneous cluster by rate of capacity.In summary, we propose ILR(Improved Load Rebalancing)algorithm. In this section, we use Matlab mathematical software and Chord software to make the simulation experiments by using the two load balancing algorithm. Then, it compare Load Rebalancing algorithm from Cumulative Distribution Function, the load of node, times of data migration and switch times of load information based on the simulation data. It can verify the correctness of ILR algorithm.