Research On Multi Stripe Repair Of Erasure Code In Distributed Storage System

In the era of explosive data growth,data is increasing at an annual growth rate of 59%.In order to achieve high availability and low storage overhead of distributed systems,various new technologies have begun to be widely used to replace the three-copy method.Many distributed systems began to use erasure coding instead of replicas to reduce storage overhead,such as HDFS,Ceph,Swift,GFS,and more.Although erasure coding can reduce storage overhead relative to replicas,it also brings higher repair costs.Compared with the copy method,erasure coded data repair needs to transmit a larger amount of data.When a node in a distributed storage system fails,it is necessary to obtain corresponding data from the remaining nodes to help repair the failed node.However,many current data repair methods do not balance the load of each node well.During the repair process,it often occurs that the amount of data transmitted by nodes with more resources is equivalent to the amount of data transmitted by nodes with fewer resources.This method does not make full use of the network bandwidth of each node,and greatly increases the cost of repair overhead.This paper takes the large-scale distributed storage system as the research object,and designs two methods to balance the load of nodes during node repair,and elaborates the related technology and theoretical basis,design ideas and test analysis.First of all,based on the unbalanced situation of reading data during stripe repair in largescale distributed storage systems,this paper designs a balanced data reading algorithm.Since random reading of data may cause too much data to be read on a node,the node that reads too much data becomes a bottleneck node.Based on the greedy algorithm idea,the data is grouped and repaired.Step 1: Select the nodes with the most participating repair blocks without repetition and group them.Step 2: After the nodes are selected,the stripes are placed by the mean square error of the number of data blocks read by the nodes.The first two steps of the cycle step will complete the set.After the grouping is completed,the data blocks required for repair are adjusted by means of the mean square error comparison.Secondly,based on the pipeline method,a load-balanced pipeline repair method is proposed to improve the problem of unbalanced computing load of nodes in the process of pipeline repairing.Prior to this,although the Comns pipeline repair method for load balancing already existed,the number of repaired data segments required by Comns to construct load balancing was very high.The Bal method proposed in this paper can achieve load balancing of each node,and the implementation method is simpler.Compared with the Basic,Cyclic,and Comns pipelines,it can be concluded that the balance of the Bal pipeline repair has been significantly improved.