Research On The Repair Pipelining Technology Of Erasure Codes In Distributed Storage

With the development of cutting-edge technologies such as blockchain,machine learning,and artificial intelligence,massive data will be produced every day,and how to store and process these data has become a hot topic in the research field of big data.The distributed storage can realize the large-scale storage requirements,but the nodes in the system often fail,resulting in the loss of storage data.In order to ensure the security and reliability of the data,the erasure codes are selected as the data fault-tolerant mechanisms to effectively repair the invalid data after the node fails.However,during the repair process,data must be downloaded from other nodes that can be used for decoding,which takes up more network bandwidth and longer repair time,resulting in the degradation of data reading performance.If the repair speed is slow,or even lower than the speed of node failure,the reliability of the system will not be maintained.Pipelining is currently the most efficient data transmission network structure,and the Repair Pipelining(RP)can significantly reduce repair time and improve repair efficiency,but there are still some drawbacks(e.g.,unbalanced load).Based on this,this topic focuses on the repair pipelining technology of erasure codes in distributed storage.Firstly,the repair pipelining based on load-balancing(NLB-RP)to deal with the problem of the unbalanced load of nodes during the repair process is proposed in the topic.The load imbalance of nodes has been improved in RP,but there is still room for improvement.In the NLB-RP,the load of nodes is balanced by adding pipelining transmission paths of different structures,and more nodes are selected to join the repair process to further reduce the load of nodes.Combining the theoretical analysis and experimental data,it can be proved that the repair method locally balances the node load,reduces the node load overall,and does not introduce the new repair costs.Compared with the RP,the load variance of nodes in the NLB-RP is calculated to be 0,which indicates that each node has the same load.Therefore,the NLB-RP is a repair method with optimal load balancing.Secondly,the partially parallel repair pipelining(PPRP)for multi-node repair scenarios in the system is designed to improve the efficiency of multi-node repair.A multi-node repair model based on pipelining path is built,which divides the repair operation into several parallel parts to improve the parallelism of the pipelining,thus reducing the total repair time.At the same time,some nodes transmit computed intermediate data to avoid redundant transmission of local data,thus reducing the overall repair bandwidth.Finally,the repair pipelining for reducing redundancy based on the piggybacking designs(Pig-RP)is proposed to extend the repair pipelining and applly it to other erasure codes.The Piggybacking designs have low repair bandwidth and low disk I/O overhead,but lack of fast repair methods.Therefore,the repair pipelining network structure is used to accelerate the repair process.First,the Pig-RP is designed based on a single-node repair,and then extended to the multi-node repair scenarios.When repairing a single node,the Pig-RP reduces repair time mainly by reducing congestion.When repairing multiple nodes,the Pig-RP not only reduces the congestion,but also significantly reduces repair bandwidth and disk I/O overhead,thereby reducing the total repair time.