Research On Erasure Coding Based Fault-Tolerant Storage Technology For Data Intensive Super Computing

As a new computing paradigm, data-intensive super computing(DISC) plays an important role in the field of high-energy physics, biology information technology, astronomy, meteorology, earthquake prediction and commercial computing. DISC is responsible for storage, maintenance and handling of data. Due to storage and maintenance of large-scale data, the system size becomes increasingly large. However, the failure of the storage node causes the increasing probability of the system fault, owing to hardware fault, operation mistake, virus attack, power failure, fire accident, etc. Therefore, the system must have high fault tolerance to ensure strong reliability and availability of the data.The high fault tolerance and space utilization ratio of erasure coding provides an effective fault-tolerant scheme for constructing large-scale storage system with high reliability and fault tolerance oriented to DISC. Nevertheless, there are many new problems need to be solved in order to use erasure coding in DISC. Firstly, the high recovery overhead of erasure coding may consume too much bandwidth. Secondly, the loss of the intermediate data created in the process of multi-stage computing will cause the job failure. Thirdly, the system may have a high energy consumption caused by the unbalance of data dissemination. To solve these issues, erasure code with high fault-tolerant ability and low recovery-overhead, fault-tolerant storage management strategy for intermediate data, and power-aware data placement strategy are proposed. The research results are listed as follows:The high recovery-overhead of erasure code may degrade the system performance. Firstly, EXPyramid, an array-based erasure code with high fault-tolerant ability and low recovery-overhead, is proposed. Secondly, two recovery algorithms, RMFA and RSFA aiming to lower the recovery-overhead, in multiple nodes failure and single node failure situation respectively are proposed. The array structure of EXPyramid enhances the fault tolerance and lowers recovery-overhead by dividing and coding data. RMFA guarantees minimum total recovery-overhead by the minimum recovery-overhead in each recovery process iteratively. Meanwhile, to find the shortest recovery-path and make recovery-overhead of single recovery minimum, RSFA uses breadth-first search strategy to find every available recovery-path. Analysis shows that EXPyramid has both higher fault-tolerant ability and lower recovery-overhead compared to current typical erasure code.Intermediate data is critical to the large-scale distributed computing of large-scale dataset. The re-execution mechanism may cause cascaded re-execution which will consume amount of various system resources. The replication strategy for intermediate data may cause large storage overhead. For these reasons, EBIDS, a fault-tolerant storage management strategy for intermediate data based on EXPyramid code is proposed. By using XOR-based EXPyramid code, which is applicable to intermediate data, EBIDS reduces storage overhead. By using pipe-line communication to computing and transferring redundant data, EBIDS alleviates the computing and communication burden of single node. Experiments show that, EBIDS create fewer redundant intermediate data than the replication strategy. EBIDS can effectively prevent the cascaded re-execution and make the system resilient for the failure with little interference to system.Appropriate placement of data in storage system based on erasure coding for data-intensive application is helpful to the load balance of storage and node utilization rate. Meanwhile, hanging up idle nodes can save energy consumption of system. On one hand, the dynamic data placement may consume too much system bandwidth; On the other hand, the static data placement doesn't consider the data access features, which may cause the node utilization rate unbalance. To solve these problems, TRBDPM, a time-relativity based power-aware data placement strategy is proposed. Considering the data access features, TRBDPM distributes the data block and redundant data block of different datasets in a cross manner to avoid the relativity during different tasks. Experiments show that TRBDPM can distribute the data uniformly, keep balanced storage load and node utilization rate, and save energy consumption by hanging up a part of idle nodes for a long time.