Research On Key Technologies Of Distributed Storage System Based On Hierarchical Overlay

With the rapid improvement in network technology, applications are changing from the traditional text and picture based on Web to more resource demanding audio and video content. At the same time, because of the remarkable improvement in network bandwith and the capacity of person computer, it becomes a hot research topic nowadays to construct a P2P distributed storage system with large-scale, low-cost, high scalability, high reliability and high performance with the PC which was ignored in the past. Nowadays P2P-based file sharing applications are characterizing a great fraction of the Internet traffic and several statistics on IP traffic have recently put in evidence that P2P traffic is starting to dominate the bandwidth in certain segments of the Internet.This dissertation makes a deep research on related technologies of P2P-based storage system. To improve data's availability, reliability and access efficiency which are deployed in the P2P storage system, We Propose a decentralized, self-organizing large distributed storage mechanism based on P2P. And designs a novel hierarchical (Distributed Hash Table) DHT routing protocol and low overhead maintenance mechanism. To improve downloading performance of large file we proposed a dynamic parallel downloading method based on hierarchical DHT.Difference from Peers which appear in traditional cluster storage systems, peers in P2P system face the new properties such as high dynamic, high heterogeneity, high asynchrony which compromise the scalability and efficiency of the whole system. Aiming at this issue, this dissertation designs a P2P-based distributed storage system HIPSS. By means of hierarchical DHT overlay which organizes large numbers of peers in Internet into a united logic network, HIPSS forms a virtual storgae pool with no single I/O addressing space out of disk provided by those peers. With the help of distributed metadata and file management, flexible replications management, efficient file searching and parallel file transferring mechanism, HIPSS builds scalable, available, efficient and massive storage system for users.Organizing data in a reasonable, effective and dependable way is a key problem to P2P storage systems. The original DHT are designed to treat peers equally. Thus yields good load balancing and robustness to failures. However, with the development of the system the weak peers can seriously compromise the scalability and efficiency of the whole system. So we present a novel hierarchical DHT routing protocol which is built on Chord but achieves better file routing efficiency in Internet by exploiting heterogeneity and proximity. This protocol exploits the proximity by grouping the data with the same properties together and distributing the data across peers within a matching cluster, which greatly boosts the local access ratio of data and optimizes the routing performance of our protocol.DHT network needs relatively higher maintenance overhead than unstructured P2P networks. As result from building and updating their routing information, p2p system generates considerable maintenance workload which not only increases the bandwidth consumption but also affects the routing efficiency. So we present here a new maintenance method to minimize the maintenance cost in a large-scale system or a dynamic environment which combines the hierarchical DHT algorithm with a peer proximity mechanism. We let storage peer in the lower layer periodically probe their successors. When a change is detected, the message will be transfered to its upper layer manage peer. Then the manage peer will broadcast this message to all its lower layer peers to update the affected routing tables rather than let lower layer peers periodically refresh their routing tables. It significantly reduces the maintenance costs of the lower ring and achieves better scalability.P2P Parallel download tools, such as BitTorrent, are very popular and scalable file distribution systems which can distribute large files quickly and efficiently. However, those systems based on tracker server relies on static IP address. Aiming at this issue, this dissertation proposed a dynamic parallel downloading method Based on hierarchical DHT. The method couples the unique ID of a tracker server with the ID of the its resident manage peer, and introduces distributed ID mapping service to implement correct route in dynamic Internet environment. By the combination of grouping peers and the locality of the downloading, the method optimizes the management of transferring and improves downloading performance greatly.