Performance Analysis And Optimization Of Erasure Codes In Distributed Storage System

Based on a completely distributed design idea,the Ceph distributed storage system eliminates the dependence on a single central node of the system and realizes unified highperformance data storage.It is an efficient solution for big data storage to build a Ceph distributed storage system using cheap common servers.However,as the scale of the system increases,storage node failures have become more common,leading to an increased risk of unreadable or even lost data.Therefore,it is necessary to design a more efficient and reliable data redundancy mechanism to ensure the security and reliability of storage data and improve the repair performance of failed nodes in the Ceph cluster.The data redundancy mechanism used in existing distributed storage systems mainly includes multiple replicas and erasure coding technology.Multi-replica technology copies data into multiple copies,and stores them in different nodes.Its low storage utilization rate is likely to cause a great waste of storage space.RS erasure codes generate redundant check blocks of data blocks based on polynomial operations,which can effectively save system storage space.However,the repairing process of a failed node using RS codes may lead to excessive disk I/O bandwidth consumption and has high computational complexity,which is not suitable for a large-scale distributed storage system.From the perspective of reducing the number of participating nodes,locally repairable code divides nodes into some local repair groups and generates local parity blocks,which effectively reduces the disk I/O of the single-node repair process.This dissertation focuses on the construction principle of locally repairable codes and compares the performance of locally repairable codes with different parameters.Aiming at the problem that locally repairable codes cannot optimize multi-node fault repair in the same local group,a locally repairable code scheme based on Clay codes is proposed.And the main research contents are as follows:(1)This dissertation studies the construction principle and optimality judgment method of locally repairable codes,compares and analyses the performance of locally repairable codes with different parameters in terms of storage efficiency,repair cost and fault tolerance,which provides a theoretical basis for subsequent system testing.(2)We study the fault-tolerant mechanism of locally repairable codes in the Ceph distributed storage system.Based on an in-depth analysis of the working principle of distributed data storage,we research the implementation and calling method of locally repairable codes in the Ceph system by studying the source code.Furthermore,we build a Ceph distributed storage system,design test cases and perform tests and analyses of locally repairable codes with different parameters in the Ceph system.Theoretical analyses and experiments have proved that compared with RS erasure codes,locally repairable codes can reduce the repair cost of the failed node,improve fault tolerance and repair efficiency at the expense of storage efficiency.Finally,according to the test results,we proposed optimization suggestions for data storage solutions based on locally repairable code redundancy.(3)To reduce the transmission bandwidth and disk I/O of the repair process of failed nodes,this dissertation proposes a locally repairable coding scheme based on Clay code which divides data blocks into local repair groups and adopts the coding structure of Clay code to encode data blocks in each local group.Performance analysis shows that the proposed coding scheme with small storage overhead can effectively reduce the repair locality while further reducing the repair bandwidth,and realize the local repair of multiple failed nodes.