Research On Key Technology Of High Reliability In Very Large Scale Storage Systems

With the development of Internet, the mount of information increases explosively, the demand for larger storage capacity, higher data availability and better I/O performance becomes more and more urgent. More and more large scale storage systems were produced and put into use. Among them, most of the storage systems are built up with a large number of Personal Computers, with superior performance and low cost. In such a system, failures are a normality which cause the data lost in the storage system. Reliability becomes more and more important with the scale up of the storage system. To insure the reliability of the system and the availability of the data, it is necessary to delve in the key technology of high reliability in storage systems.This paper firstly introduces a high reliable storage system architecture, then studies the self-adaptive high reliable data layout, small low density parity code(LDPC) which is adapt to fault tolerance in storage systems, failure prediction based data recovery mechanism(PBDR).(1) Replication placement concerns the redundancy and fairness of the data layout in very large scale storage systems. Aiming to system reliability and data availability of replication layout in storage system, Integer Programming is used to describe the reliability hierarchy oriented data layout optimization in storage systems, and this work is NP-hard. Then a high efficient data layout algorithm based on greedy algorithm is designed, this algorithm can minimize the reliability cost of all storage nodes in the system. Also this algorithm can optimize the fairness of the data layout.(2) Aiming to optimize the multi-replication data layout scheme of the storage system, based on markov model, variable rank data layout model(VRDL)-a model to evaluate the reliability of the storage sytem is designed. VRDL can evaluate a series of factors'the impact on storage system reliability. Thus VRDL can help in design of high reliable storage system. The factors include the replication rank, the average size of data object, the system scale, the capacity of storage nodes, the average failure time of storage nodes and the latency of failure detection.(3) By application of erasure code theory in the strategy of storage node failure in very large storage system. We establish a erasure code model in multi-storage nodes environment. Also, we proposed a small LDPC code scheme in multi-storage nodes failure environment. This code scheme can insure not only high availability but also better performance than that of RS code.(4) As improvement of the capacity of hard disk is faster than that of hard disk I/O bandwidth, the data reconstruction time of a failed disk becomes longer and longer, Which widen the weak windows time of the storage system. Thus, the possibility of data loss in storage system is increased. We discuss the data recovery mechanisms of three data redundancy which include two-way mirror mechanism, three-way mirror mechanism and RAID5 mirroring mechanism. Further more, we propose a prediction based data recovery mechanism (PBDR). By exploiting the idle storage node resources, PBDR can start the data reconstruction process before storage device failure happening, thus improve the reliability of the storage system.