Application And Research On Key High-reliability Technology In Mass Data Storage Systems

With the development of information technology, data is increasingly becomingan important resource in our daily lives. According to the statistics, the total amount ofdata which the world created in2010, stored and replicated has reached1.2ZB, and in2011the amount is1.8ZB. It is expected to reach nearly8ZB in2015. The primaryproblem brought by the explosive growth of data is the increasing costs of storageequipment and management. Now, the scale of storage nodes in the modern data centeris ranging from tens of thousands to hundreds of thousands. Disk and storage nodefailures have become a normal behavior in the huge storage systems. At the same time,the network connection device and other components of storage node often becomeinvalid. In order to meet the expanding demand of reliability, availability, and otherrelevant characteristics of data storage, there need new techniques to cope with thecurrent situation. How to implement the mass data storage technology with lowredundancy and high reliability has become a huge challenge.Thus, this dissertation focuses on the key technology of constructing the massdata storage system with low redundancy and high reliability. Basing on the longresearch of the theory of data reliability and the framework of the storage systems, wehave made some progress. The main contents of this dissertation are as follows:1. As the RAID technology has already been widely used in the data storagesystem, we propose a new level array code based on the exclusive-OR operation:EX-ENOD code. This code can tolerate any random three failures and it has the beststorage efficiency. According to the geometrical configuration of the code, we proposea decoding method with low computational complexity. Further more thecomputational complexity of this method is much lower than those of other erasurecodes. Meanwhile, this decoding method can be applied to the STAR code, the EEODcode and so on.2. In order to improve the reliability of the mass storage system, we design andimplement Vandermonde system encoding method that is based on {0,1} symboldomain in the mass storage system. This encoding method inherits the excellent features of Vandermonde code that is on the traditional finite field. There is no need toconsider the limits number of the storage nodes and fault-tolerant parameters. At thesame time, it can achieve the best storage efficiency. This method can avoid thedisadvantage of the traditional encoding on finite field which requires a lot of queryoperations. In particularly, this method is very suitable for the binary representation ofthe data in the computer system, so it is more suitable for the mass data storage thanothers.3. According to the distribution of the "1" element in each row vector of theencoding matrix in {0,1} symbol domain, we propose an optimization algorithm whichcan reduce the computational complexity of encoding and decoding. At the same time,In order to reduce the high bandwidth that is needed in the traditional decodingreconstruction process, we propose a decoding method which is based on the paritycheck matrix. We also give a low-bandwidth reconstruction method according to thecharacteristics of the parity check matrix column vector. This low-bandwidthreconstruction algorithm can be extended to all the encoding storage systems that arebuilt on {0,1} symbol domain.4. According to the characteristics of the data layout of erasure codes, we design amass storage system infrastructure with low redundancy and high reliability. Thissystem combine the data deduplication technology and encoding technology together.Basing on the structural characteristics of the erasure code, we extend an energy-savingand self-adaption access strategy to the storage system. At the same time, we alsointegrate the data redundant elimination and data validation together in the system.