Optimization Of Heterogeneous Storage Mechanism Based On Ceph Erasure Code

Recently,the explosive growth of data volume has posed a huge challenge to the traditional storage mode.In order to meet the rapid development of storage requirements,distributed storage arises at the historic moment.Among numerous distributed storage system,Ceph is widely used because of its open-source and active community as well as unified support for storage service.However,Ceph gains more storage space with the cost of encoding and decoding calculations in erasure code backup mode and does not perceive the heterogeneity of the underlying storage devices.It will lead to the low read-write performance and further aggravate the load difference among OSD nodes,which could not provide efficient and reliable storage services for users in high load scenarios.Therefore,the present study explored the optimization storage mechanism based on Ceph erasure code,and proposed corresponding strategies to optimize its performance.The specific research work were as follows:1.The optimization strategy of heterogeneous hybrid storage based on Ceph erasure code was studied.When Ceph selects OSD node in erasure code backup mode,it only takes the storage capacity as the only reference indicator,and does not consider the factors such as computing power,storage load,and storage media of the OSD node itself,which decrease the Ceph system performance.To address this issue,we proposed the optimization strategy of heterogeneous hybrid storage based on Ceph erasure code.It use TOPSIS to model and solve the target parameters to obtain the optimal OSD node.Comparing with the Ceph native node selection strategy and LCMS strategy,experimental results suggested that the optimization strategy proposed in this chapter could effectively improve the read and write performance of Ceph system in the erasure code backup mode.2.The load balancing strategy based on Ceph erasure code was studied.Since the existing Ceph task allocation strategy does not consider the load balance among OSD nodes,a large number of erasure code reading and writing tasks may be allocated to nodes with higher load but relatively large storage capacity for processing,resulting in poor performance of Ceph under high load scenarios.Based on this,we proposed the load balancing strategy based on Ceph erasure code.This strategy considered multiple load parameters from the OSD node dimension,and established an OSD load model.Through the experiment comparison with Ceph native task assignment strategy,LCMS strategy and the optimization strategy of heterogeneous hybrid storage based on Ceph erasure code,results showed that the load balancing strategy proposed in this chapter can significantly improve the performance of Ceph clusters in high load scenarios.3.System design and implementation were carried out according to the strategy proposed in this thesis.System information collection and visualization were completed through open source platforms such as Prometheus and Grafana,and system modules such as OSD load module,closeness calculation module and closeness adjustment module were realized.In addition,the system implementation was verified using the open source Cos Bench microbenchmark object storage and test platform of Intel coropration.The research work of the thesis shows that the strategy proposed in this thesis can effectively improve the read and write performance in Ceph erasure code backup mode,and make Ceph to better adapt to high load scenarios,which has good practical significance.