Research And Implementation Of The Erasure Code For Cold Data In Distributed Storage System

Since the birth of the Internet, especially in recent years, mobile Internet, the Internet of Things and cloud computing, led to the exponential growth of network data. This caused great pressure to the storage system. To ensure the reliability of data, storage systems commonly store data with replication, which causes a large number of data redundancy. Using erasure code for data fault tolerance can reduce data redundancy, but data availability will be compromised. This has resulted in some degree of conflict.Internal data storage system is divided into cold data and hot data according to frequency of use, then the cold data is stored by way of erasure codes. This ensures data availability while greatly reducing the data redundancy of the system. According to computing power and storage space the physical nodes are divided into multiple virtual nodes and virtual nodes dispersed to a hash ring by consistent hashing, to construct a fully symmetrical, hash to the center of the ring, in order to achieve a uniform data distribution and task scheduling. Encoding and decoding process by using Bit-Matrix technology reduce the number of 1 in coding matrix, which effectively reduces the encoding and decoding process XOR operation times. At the same time, the introduction of Schedule techniques, reducing encoding/ decoding process steps by effective planning, especially in the decoding process subsequent decoding operations before the calculation of the effective use of data, effectively reduces the computational complexity and speeds up the encoding/decoding speed.Achieve the above design and test the system using the IO testing tools. Test results show that, by introducing a mechanism erasure codes, write speed has been greatly improved as opposed to the traditional mechanisms of copies. For 4: 2 configuration of erasure code, as opposed to the traditional 3 copies of policies, storage efficiency is improved by 100 percent. When the memory block is 1024 k, sequential write speed increases of 106%. Of course, when the memory block is small, the write speed is limited, for example, when the memory block size is 4k, sequential write speed increased by only 6.5%.