The Research Of Tiered Erasure Coded Distributed Storage System

With the advent of the era of big data, the security and reliability of data storage is becoming increasingly important. Existing distributed storage systems often use replication to ensure data reliability, and in order to improve storage efficiency, those systems will apply erasure code technology, which however has brought enormous repair costs, because it requires multiple disks reads and network transfers for reconstructing an unavailable data block. And the frequency of data accessed in the storage system is not the same, existing systems will use a higher performance storage media to save data so as to improve the access speed of hot data. However, when using erasure codes on hot data, the repair will seriously affect the performance; Use the replication on cold data will seriously affect the storage space usage.In order to take into account the data reliability and system performance,the encoding method optimization for erasure code based distributed storage system and tiered storage had become hot topics of academic research.This paper design and implement a tiered erasure code based distributed storage system DHFS(Distributed Hierarchical File System). For reducing data repair costs in the existing coding methods, it provides a new encoding method and different encoding parameters to ensure the data reliability and the performance of data repair and degraded read. While for the cold and hot data, this system provides a tolerant scheme in the hard disk and memory. When hot data is lost, it can quickly read data from other nodes in the memory and reconstruct the source data, which improves the data repair and the degraded read performance of hot data.The test results show that the system achieved a relative increase of storage efficiency by about 25% compared to triplication and repair speed relatively increased by about 25% compared to the RS and CRS coding method with the same parameters; In tiered storage environments, the read speed is 2 times faster than the traditional cache structure and the repair speed is almost the same. The test results indicate that in the aspects of support and optimization for the existing erasure code, the present system saves more space than the replication based storage system, has a faster speed of data repair and better degraded read performance than the original coding methods and also take the advantages of tiered storage architecture to improves the data access and recovery performance of hot data.