Research On Data Placement Mechanism Of Cloud Storage System

With the rapid development of modern society,the growth of massive data is so rapidly,and this greatly stimulates the demand for storage.With the development of computer tech-nology,cloud storage,as a high-capacity,efficient,economical and reliable storage technology,has gradually become a key basic technology in the field of computer storage.Academics and industry researchers have invested a lot of research in this field,resulting in a lot of scientific re-search results and commercial applications.However,cloud storage technology also faces many technical problems and challenges that need to be solved urgently.This thesis focuses on three kinds of challenges:low resource utilization of cloud storage system,unbalanced load with-in the system,and difficult decision-making of system construction schemes.These problems and challenges are based on the requirements of different application environments,and cloud storage systems are divided into private cloud storage system,public cloud storage system and hybrid cloud storage system.In view of the problems and challenges faced by these three types of systems,based on the existing related work and achievements,this thesis focuses on a series of issues,such as optimization of system resource utilization,identification of abnormal heat data,data relocation of overload equipment,system design decision-making and so on.By adjusting the data layout to improve the utilization rate of system resources,locating and re-distributing abnormal data to improve the system load balance,and using economic models to assist the design and decision-making of the system,the problems mentioned above can be solved accordingly.The above research directions and contents are interrelated and interacted,but the ultimate way of imple-mentation is around the data layout mechanism.The main research contents and innovations of this thesis are summarized as follows:The goal of data placement research in public cloud storage system is to make the device load in the system more balanced.Public cloud storage system is a large-scale heterogeneous system,in which the massive data layout is not uniform,and more difficult is to achieve data access and storage utilization at the same time.In the system scale with massive data and large equipment,the general optimization model can't deal with the data layout problem because of the high complexity of its corresponding management algorithm.Firstly,the classification algorithm is used to classify the data and equipment in the system to solve the layout equi-librium problem of different data.Secondly,the improved counting Bloom filter algorithm,MlCBF(Multi-level Counting Bloom Filter),is used to locate and identify the heat data.Finally,the improved consistency hash algorithm,multi-selection,is adopted.The consistency hash al-gorithm(McCH,Multi-choices Consistent Hash)is chosen to relocate the abnormal data,which provides a feasible solution for a series of problems faced by public cloud storage system.The goal of data placement research in private cloud storage system is to optimize system resource utilization and reduce system cost.Because private cloud storage systems are mainly used in enterprises,their main characteristics are stability,low cost,large capacity and low throughput performance.This thesis presents a resource optimization model for private cloud storage system,and uses heuristic algorithm to solve it.Finally,under the limited resource conditions,the utilization rate of disk resources of the system is improved,the number of devices used in the system is reduced,and the overall cost of the system is reduced.The research on data placement strategy of hybrid cloud storage system aims at realizing the design decision of hybrid cloud storage system.In real business applications,some enterprises choose to build private cloud storage systems by themselves,while others choose to rent third-party public cloud storage services.More enterprises use hybrid systems of public cloud storage services and private cloud storage systems for practical needs.No matter what form of cloud system is adopted,enterprises hope to achieve the lowest construction and use costs on the premise of meeting the needs.However,due to the difficulty of horizontal cost evaluation of rent and purchase,it is difficult to determine the mix ratio of rent and purchase,which is the key problem to be solved in the design of hybrid cloud storage system.Based on a financial theory,Net Present Value(NPV),this thesis proposes a design model which can realize the comparison of horizontal evaluation between rent and purchase.Then kNN and K-means algorithms are used to classify different data and devices,so that different data can be more reasonably distributed in the corresponding system.Finally,the mixed proportion of decision-making system can be quickly and effectively,and the design requirements of enterprise hybrid cloud storage system can be achieved at the least economic cost.