Research On Hybrid Key-Value Big Data Storage System

The rapid development of Internet technology has led to the explosive growth of the amount of data in the world,there have been a variety of structural types of massive data,that is,we now often say that the big data.Due to the big data contains huge amount of information and change speed characteristics.Therefore,big data storage system must be capable of rapid data storage,otherwise due to aging,is likely to lose valuable data information.Because the traditional storage strategy is difficult to meet the needs of big data storage,so it is necessary to study the new big data speed characteristics of big data fast distributed storage and access policy.Currently popular distributed large data storage system based on key-value data model is used to store data on disk in a single node.Disk's low speed I/O will become the bottleneck of the entire system,it is difficult to meet the requirements of big data storage and access to fast.And a new storage device solid-state disk(SSD)has a lot of read and write performance than the disk.At the same time,solid state disk also has a trend of prices are getting lower and lower,capacity are getting more and more large.So it is an important research topic in the field of big data technology to combine the solid state disk and the disk to form a hybrid architecture to be the persistent storage of the single node.An in-depth analysis and research on the structure of the hybrid storage system based on solid state disk is carried out in this paper.This paper puts forward a new realization method of hybrid key-value data storage system based on solid-state disk and hybrid key-value data storage engine.The research methods proposed in this paper can provide fast data reading / writing,backup and recovery storage services for big data storage system.The main research work in this paper is as follows:1.Study and propose a novel fast hybrid key-value big data storage system(HKV-BDSS,hybrid key-value big data storage system)based on the solid state disk.The hybrid key-value storage engine on single node uses SSD as secondary storage disk,as well as be a cache between memory and disk.At present due to solid-state disk capacity are getting more and more large,so a large number of key-value data can be buffered in solid state disk,greatly improve the single node throughput and reduce the service delay,then improve the access speed of the whole system2.Study and design a distributed key-value access system.Because the data is eventually saved on many machines,so in order to facilitate management of the cluster machines,this paper developed distributed key access system based on the cluster.The distributed key-value access system in this paper uses a combination of Master/Slave structure and P2 P ring structure way architecture.The architecture combines the former with the advantages of central node,can manage cluster nodes conveniently and simply;and combination of the advantages of the latter can use distributed hash algorithm for data convenient storage and less data migration.The architecture can solve the problem of scalability,high availability,data consistency and load balance of distributed systems.3.This paper describes the design and implementation of most functions of hybrid key-value big data storage system.And through the deployment of the experimental environment,test the storage engine read and write performance on single node and test the distributed key-value access system core function.The experimental results show that the design has reached the intended target.The design idea proposed in this paper has a good reference value for the further research of large data storage system.