The Design And Implementation Of Disaster Tolerance Cloud Storage System Based On Pyramid Erasure Code

With the rapid growth of information today,there is no doubt that information means value,people focus far more on how to safely and reliable save information.Cloud storage system has been widely used for its high performance,high scalability,high availability,high capacity and so on.Usually,cloud storage system use both of three mirror copies mechanism and erasure code mechanism to to ensure the reliability of the data it keeps.As for three mirror copies mechanism,when the three copies of the same data block lost at the same time,the interrelated file must has been damaged,howerver,if we increase the backups which must reduce the utilization ratio of storage space in the cloud storage system.And,the RS erasure code,which has been used in the cloud storage system,due to the encoding characteristics of itself,the number of blocks to be read in the decoding from fault recovery is constant,if we can reduce the number of blocks to read when recovery from less blocks lost,it can not only accelerate the decoding speed of recovery,can also reduce the failure recovery bandwidth usage so as to improve the overall performance of the cloud storage system.Just from this point of view,this paper has carried on the thorough research,finally the disaster cloud storage system which is based on Pyramid erasure code is designed,the main work of this paper is summarized as below:First of all,without adding extra storage space,this paper improved the mechanism of three copies by using only XOR parity data blocks,and called it high fault tolerant three copies mirror mechanism.When original data block's three backup blocks lost at the same time,it can also recover the losted original data block,thus compared with the original three copy mirror mechanism,high fault tolerant three copies mirror mechanism's fault tolerant ability is much better,and at the same time,the parity blocks only using XOR to calculate,which makes the implementation of this method is relatively simple.Secondly,this paper make Pyramid erasure code used in the cloud storage system,replacing the RS erasure code which has been used to,thus can effectively reduce the amount of the data to read in the recovery of single block fault decoding,accelerate the data read speed,and reduce the amount of recovery calculation.Along with the thorough research to the erasure code technology,Pyramid erasure code is improved in this paper,and a preliminary coding scheme is proposed,which reduces the amount of data transmission between the data storage center when recovery from two blocks failure and accelerates the data read speed,and the improved Pyramid code keeping coding structure with the original Pyramid code which increase the fault recovery way of decoding.Finally,based on Hadoop distributed storage system,using the improved three mirror copies mechanism and Pyramid code,this paper designed the Disaster Tolerance Cloud Storage System.The Disaster Tolerance Cloud Storage System use the improved three copies mirror mechanism coding and saving thermal data files,and code and store the cold data files using the Pyramid erasure code.According to the user's attention to the files,the files are divided into cold data and thermal data,and regularly scanning the files to judge its status,if find the status of the file changed,then we use proper method to change this file's coding scheme,in order to ensure the reliability of this file on the premise,and as far as possible to improve the availability of this file.Finally,this paper perform the functional test for the cloud storage system prototype,and respectively analyze the improved three copies mechanism and Pyramid erasure code's decoding recovery performance,the results all have reached the expected goal.