Research On Backup And Repair Technologies Based On Erasure Codes In Distributed Storage Systems

The distributed storage systems connect multiple storage nodes by networks,and can realize the storage for massive data.This kind of system generally uses the commercial server nodes to work,and the frequent failures of the nodes make the data reliability be low.Erasure codes generate redundant data by encoding the original data so as to enhance the data reliability.However,when erasure codes are applied into the distributed storage systems,the efficiency of data insertion is low,the repair cost is high and the repair time is long.These problems are researched in this dissertation,and the contributions are as follows:Aiming at the problem of low data insertion efficiency under the traditional erasure coding scheme,a balanced data distribution method for data insertion is proposed to reduce the insertion time and increase the throughput of the insertion process.The basic idea of this method is that each data block is split into a smaller size,and the transmission capability of the intermediate nodes is used to transmit the data to achieve the effect of parallel transmission.Then the data transfer scheme is modeled into a quadratic programming problem,and the optimal data split solution can be achieved by solving the minimal insertion time.Experiments show that this method reduces the insertion time and increases the throughput of the insertion process and improves the efficiency of data insertion.Aiming at the problem of high overhead in data repair,a new type of composite code,called IZSHC,is proposed.Using IZSHC,to effectively reduce the data read amount from disk and data transfer amount during repair process.The basic idea of building IZSHC is to get the minimal IZS encoding group by decomposing the structure of duplication code for IZS code,and the basic features of the data repair cost of IZS encoding group are fused into the hierarchical code,and its layer structure is used to extend the IZS group,and the number of nodes participating in the repair process is reduced.This method combines the low traffic of basic IZS code with the hierarchical grouping pattern and reduce repair overhead.Experiments show that IZSHC reduces the number of nodes involved in the data recovery,so that the total disk overhead and the total transmission bandwidth overhead are reduced.Aiming at the problem of long time for repairing single node,a collaborative repair method of multiple tree is proposed,so as to reduce the repair time for single failed node.The basic idea of this method is that cooperatively repairing the single data block by organizing more bandwidth into multiple trees,each tree repairs one part of the single data block,so as to quicken repair.Then increase the bandwidth of each tree by iterative edge exchange,so as to fully utilize the bandwidth.Experiments show that the repair time for the single failed node is reduced,compared with the single tree repair method based on the network encoding.Aiming at the problem of long time for repairing multiple failed nodes,a repair method for multiple trees based on MSR code is proposed,so as to reduce the node average repair time.The basic idea of the method is to construct a tree minimal maximum bottleneck spanning tree for each failed node in turn;when constructing one tree each time,under the premise that the bottleneck bandwidth in the current tree is not reduced,other large bandwidth edges are left to the construction for the next tree,so as to increase the probability of achieving the bottleneck edge with larger bandwidth for each tree.This method considers the relevance for bottleneck bandwidth for each tree,and fully utilize the bandwidth of the network.Experiments show that,the node average repair time is reduced,compared with the existing multiple node parallel repair method.