Exploring Optimization Techniques For Distributed Object-based Storage Systems With Heterogeneous Storage Devices

With the development and popularization of applications such as e-books,e-commerce,medical big data,genetic engineering,high-performance computing,virtual/augmented reality and streaming media,digital information has seen siginificant growth in recent years.The storage and processing of such information has pushed the limites of the underlying storage systems in terms of capacity,performance,reliability,and scalability.The traditional disk-based centralized storage system cannot effectively solve the scalability problem of big data applications.Since the capacity,volume and power consumption of the disks become the bottleneck that restricts system expansion.In recent years,the rapid development of new types of storage media,such as non-volatile memory,flash memory and archiving hard drives,has many advantages over traditional disks,such as high density,low latency,and low power consumption.The development of heterogeneous distributed object storage systems based on hybrid media has been proposed,which is an effective method to reduce data center cost.The introduction of these new media brings new challenges to the traditional storage structure.Traditional system software and application design concepts need to be adjusted in order to adapt to the characteristics of the new media.Therefore,this paper first analyzes the basic architecture and research hotspots of hyterogeous object storage systems with hybrid storage device.Based on these research backgrounds,this article discusses three optimization strategies based on hyterogeous object storage systems with hybrid storage media: The management of non-volatile memory in distributed object storage systems,the data distribution algorithm for object storeage system and the data placement strategy in Mapreduce framework.The main research content of this article is as follows:(1)NVM management strategies in object storage systems with hybrid media.For a single node deployed with non-volatile memory,we have designed a new memory management scheme to improve the wear-leveling effect of non-volatile memory-based file systems,thereby increasing the lifetime of non-volatile memory devices.The basic idea of this method is to select a memory page with a relatively small number of writes each time a memory page is allocated,and establish an index structure by utilizing DRAM to speed up the lookup operation.(2)Research on data distribution algorithm for object storage system with hybrid media.The traditional hash-based data distribution strategy can ensure that the data is evenly distributed to different types of storage devices.However,the disadvantage of these algorithms is that they do not consider the heterogeneous characteristics of storage devices.For example,flash memory and non-volatile memory are limited in endurance and their service time is shorter than that of ordinary disks,and therefore are not suitable for storing write-intensive data objects.This paper designs a new data distribution algorithm to fully consider the heterogeneous characteristics of storage devices and the reliability,stability,scalability,and read/write performance of the system.The core idea of this algorithm is to use Biased-Hashing to dynamically select the appropriate storage device according to the characteristics of the data object and solve the problem of the traditional uniform hash algorithm in heterogeneous storage systems.(3)Research on data placement strategy for MapReduce framework based on heterogeneous environment.In a cluster with hybrid storage devices,we find that the Map/Reduce task execution time is different due to heterogeneous devices.We designed a new data placement algorithm based on the Ceph architecture to consider the load balancing among clusters and heterogeneous features to minimize the execution time of workload.