Research On Data Reconstruction Strategy Of Fault-tolerant Storage System Based On Erasure Codes

In the era of big data,the scale of data has expanded dramatically,and huge storage pressure has led to frequent failures of nodes in distributed storage systems.As an important means to ensure data reliability,fault-tolerant storage is very important.At present,fault-tolerant technologies mainly include replication technology and erasure coding technology.Erasure codes have attracted widespread attention because they have the same fault-tolerant capability as replication and are lower than the storage overhead of replication.However,the existing serial encoding and decoding algorithms of erasure codes have obvious bottlenecks and cannot reasonably utilize system resources,resulting in lower overall efficiency of erasure codes.In addition,the existing data reconstruction process of multiple failed nodes requires a large amount of data to be transmitted,which consumes a large amount of network resources,and has become a major factor limiting the improvement of erasure coding performance.In view of the above problems,this paper studies the encoding and decoding technology and the reconstruction process algorithm,and proposes corresponding improvement schemes to optimize the data reconstruction efficiency in the fault-tolerant storage system.The research work of this paper is divided into the following two parts:(1)A parallel encoding and decoding algorithm based on cache queue(PEDACQ)is proposed.PEDACQ aims at the problems such as low parallelism and I/O bottleneck limiting coding efficiency in the process of serial encoding and decoding.By maintaining read cache queue and write cache queue,and using multi-threading technology,it tries to reduce the impact of I/O bottleneck as much as possible,so as to improve the efficiency of encoding calculation and accelerate the encoding and decoding process.Finally,through experiments and comparison with traditional serial encoding and decoding methods,the results show that the scheme has higher encoding and decoding efficiency.(2)A parallel reconstruction method based on network topology(PRNT)is proposed.PRNT addresses the problem of large network resource overhead caused by data reconstruction in the case of multiple node failures.First,a parallel reconstruction set is constructed to determine whether the data object that each failed node belongs to is the same.The failed nodes from different data objects are added to the set to ensure the maximum degree of parallelization of node reconstruction in the set.Then for each failed node in the set,this paper proposes an optimal reconstruction tree algorithm based on the network topology.The algorithm first takes the newcomer as the root node and the surviving node as the provider node to construct a minimal reconstruction tree.Based on the idea of regeneration codes,as many provider nodes as possible are added to the reconstruction tree to reduce the total amount of transmitted data,thereby improving the reconstruction efficiency.Compared with the traditional star and tree reconstruction methods,the experiments show that the method has lower reconstruction time,higher reconstruction success rate and data availability.