Research On Decentralized Data Distribution Algorithm For Distributed Storage System With Heterogeneous Devices

To ensure the high scalability of the storage system and the high reliability and strong consistency of the data,cloud storage manufacturers generally use CRUSH,which duplicating and writing Synchronously on the link with multiple replicas.However,with the iteration of SSD products,there are plenty SSD storage devices with different interfaces,different manufacturers and different capacities in the distributed storage system.The degree of differentiation of data copy storage devices directly affects system service performance.A reasonable data distribution algorithm can directly improve storage system performance and improve storage device efficiency.In view of heterogeneous storage system,two data distribution algorithms are proposed in this paper.One is the decentralized data distribution algorithm based on performanceaware storage pool(pCRUSH),which can sense the performance of storage devices and partition heterogeneous storage pools.To reduce tail delay of the storage system,different copies of the same data should be put in storage devices of the same performance partition.The other is the load-aware decentralized data distribution algorithm(LLPC),is proposed to further improve storage resource utilization.LLPC performs feature perception on upperlayer data I/O,by recording linked list with hot key and using hot data mapping algorithm.The results show that the tail delay of the storage system is reduced by about 43.77%,while applying pCRUSH instead of the CRUSH.The average write latency of the storage system is reduced by about 79.70%,by combining LLPC and pCRUSH;At the same time,pCRUSH improves storage system IOPS performance by 56.52%?66.80%.The IOPS performance of distributed storage systems can be improved by 2.29X?2.79 X,while associating LLPC and pCRUSH under random read and write loads with hotspots.