LEAP: Load-Aware Encoding Of Replicated Data Into Erasure-Coded Data

Erasure code is increasing used in distributed system for storage optimizing since they provide higher reliability and significantly lower storage costs than data replication with trade off on recovering and reading performance. The following three factors lead to the study of encoding procedure of heterogeneous distributed storage cluster.(1) For 24-online system, the normal jobs of users can't afford noticeable damage by decoding.(2) When the cold data lay out with irregular way in the cluster, large number of blocks encoding on the same node. Slowest part is “wooden barrel effect”.(3) Heterogeneous of cluster caused by different hardware, different pattern of user I/O and random placement of data. The locality based schedule model can't solve the problem of load imbalance caused by heterogeneous.The contributions of this paper are summarized as follows:(1) It proposed a load-aware encoding schedule scheme on heterogeneous distributed storage system.(2) It developed an capacity predict and feedback dynamic calibration method to accurately use the resource of storage nodes on encoding process.(3) It quantitatively evaluated the impact on user jobs of two encoding schedule schemes for replication storage clusters.Experimental results show that this method can be greatly accelerated cooling copies of data blocks to the encoding process of erasure codes, as opposed to the traditional scheduling method based on data locality speedup can reach 2.78; at the same time on the user's response time is reduced by 6.89%.