| In the era of big data,the scale of data is growing,which puts strict demands on the economic cost and fault-tolerant efficiency of storage systems.The system should not only have extremely high data access performance,but also be able to recover quickly when encountering faults.In large-scale distributed storage systems,single unreliable component failure is very common.Generally,storage systems based on erasure codes have low storage overhead and high fault tolerance,but recovery in case of failure will lead to serious cross-rack bandwidth loss.Moreover,the nodes participating in the recovery are random,and if the node state changes instantaneously,it may lead to transmission interruption,thus affecting the recovery efficiency and wasting additional resources.In addition,there are differences in access frequency of files in distributed systems,and sequential recovery will cause some frequently accessed files to be unable to recover quickly in the first time,resulting in low availability of files and affecting users’access.Aiming at the above problems,this paper optimizes and improves the fault-tolerant strategy based on erasure codes in distributed storage.The main work is as follows:(1)Cross coding layout based on Reed-Solomon(HV).In order to solve the problems of large amount of cross-rack and cross-node data transmission and occupying more network bandwidth caused by data recovery of distributed system based on erasure codes storage,a cross-coding layout(HV)based on RS is proposed.HV architecture adds vertical parity calculation based on traditional RS erasure codes storage,uses RS codes with different parameters,and stores the results of vertical parity calculation in the current node,which can be decoded and recovered from the node itself in case of node failure,thus reducing cross-rack data transmission during recovery,thus reducing network bandwidth occupation,and improving recovery efficiency.Through experimental analysis,HV can reduce the cross-rack bandwidth of RS and D~3 by 55.56%and 33.33%at most.When a single data block is lost,the cross-node bandwidth can be reduced to zero to the greatest extent.Compared with other erasure codes strategies,many cross-rack data transmission is reduced,and the advantages of HV are more obvious with the increase of data volume.(2)Supply node selection strategy based on distance and load(SNSP).Aiming at the problem that the existing recovery algorithm randomly selects the supplier nodes to participate in the recovery,which is random and easy to cause the recovery time to be greatly prolonged due to the long response time of the nodes,this paper proposes a supplier node selection strategy based on distance and load(SNSP).According to the network distance and node load,the supply node is selected to reduce the recovery time and improve the recovery efficiency.SNSP and HV layout are integrated to form HV-SNSP system.Experiments show that the recovery time of HV-SNSP is greatly reduced compared with RS(3,2)and D~3,and the recovery time can be reduced by up to 42.41%and 33.49%respectively when a data block is lost.When multiple data blocks are lost,the recovery time can be reduced by up to 41.47%and 36.59%respectively.When the whole node fails,the recovery time can be reduced by 23.96%and 19.08%at most,which greatly improves the recovery efficiency.(3)Recovery Priority Decision Strategy based on block heat(RPDS).Aiming at the problem that the existing strategies restore data blocks in random order without considering the access frequency of the files where the data blocks are located,so that many file access requests cannot be satisfied normally before the recovery is completed,thus reducing the availability of files,this paper proposes a recovery priority decision strategy(RPDS)based on block heat.By prioritizing the files and data blocks to be recovered and adjusting the recovery order,hot data blocks can be recovered first and accessed by users,thus improving the availability of the system.Experiments show that the RPDS strategy can improve the access success rate by 76.83%compared with RS at the same time when the data block is partially lost.Similarly,in the case of a single node failure,RPDS can improve the access success rate by up to 50%,which greatly improves the availability of the system. |
