Design And Implementation Of Highly-Available Distributed Storage System Based On Erasure Code Technology

After the 911 accident, data storage becomes more and more critical. It becomes a focus how to ensure the security of the data, which desires the appearance of the safety and trusty distributed storage system. It can distribute data on the network or several nodes and make user obtain the data in the continually-effective and highly trusty way. Even with nodes failing, network disconnecting, or suffering hostile attacks, the system can still support the data storage service.After the introduction of the distributed hash table, it becomes a hot topic nowadays the wide-area storage system based on the peer-to-peer. The projects being carried out currently include OceanStore(Berkeley), CFS(MIT), Past(Rice&Microsoft), Granary(Tsinghua) and so on. But because of such characters as heterogeneity, dynamic, distrust, and vulnerability in the real P2P environment, the availability of the storage system is decreased. Based on the research of the 863 project"cooperative emergency-corresponding and drift-based survivable system research"and combining the erasure code and DHT, this dissertation studies the design and implementation of the highly-available distributed storage system. The main contributions are as following.Firstly, this dissertation analyses the advantages of the P2P technology in the distributed storage system and summarizes the current research actualities. It also studies the relational theory of P2P technology.Secondly, the dissertation analyses and implements the erasure code algorithms based on the Vandermonde matrix and the Cauchy matrix and make a comparison between them. The result shows, compared to the algorithm based on the Vandermonde matrix, the encoding efficiency of the algorithm based on Cauchy matrix increases by 43 percent and decoding efficiency of that increases by 76 percent.Thirdly, this dissertation proposes a highly-available distributed storage system based on the erasure code and analyses in detail the file en-decoding module, block distributing and fetching module, dynamically maintaining module and the other function module, which make up of the system. And it also analyses the system performance.Fourthly, this dissertation implements the HHStore, the prototype of the highly-available distributed storage system based on the erasure code. The test of the system performance shows that the downloading time of the central server increases very fast, but the performance drops fast when the number of the nodes increases rapidly while the HHStore can keep excellent performance even when the network size becomes very large.From what has been done above, the dissertation designs and implements the highly-available distributed storage system based on the DHT and erasure code. The system can download and store data safely and trustily. And it also can satisfy the distributed storage demand of the secrete data in such key departments like army and the parallel requests from the numerous nodes in wide-area storage system. It has the better availability, security and manageable and fits for all kinds of data storage.