Design And Implementation Of Hierarchical Hybrid Storage System Based On Consistent Hashing

In big data era,how to store the massive data efficiently,with very large capacity and in low cost way is the key for storage system.According to the law of 80/20,in most time,80% of the requests access to only 20% of all data.That is,the most requests focus on the small file and hot data.Hard Disk Drives(HDD)and erasure policy are suitable for storing big and cold data.On the contrary,Solid State Drives(SSD)and replica policy are suitable for storing small and hot data.Therefore,there is a trend that the data with different characteristics has their own storage device and storage policy.Based on OpenStack Swift Object System,this paper takes different storage devices and storage policies into consideration,and designs a hierarchical consistent hash ring to build hierarchical hybrid storage system.In this system,we create SSD Ring and HDD Ring respectively.The SSD cluster chooses replica policy,while the HDD cluster chooses erasure policy.According to the importance,access frequency and capacity,data choose relative hierarchy to be stored.Furthermore,we introduce the monitoring system to determine the frequency of data collection and migration threshold in different time.Monitoring system monitors the utilization factor of the SSD cluster.When the utilization factor exceeds the threshold,we move the cold data which is in SSD cluster to HDD cluster according to access times and the last visit time.Moreover,we introduce the warm data to reduce unnecessary data migration caused by accident.When data in HDD cluster are accessed frequently in a period of time,the hotrank flag would be upgraded from cold to warm or warm to hot,data which hotrank flag is hot will be moved to SSD cluster.The experimental results show that the proposed hierarchical hybrid storage strategy according to data's characteristics enhances the system flexibility,improves the read and write performance and ensures the stability.