Design And Implementation Of Distributed Key-value Storage System

With the rapid development of Internet and the wave of Mobile Internet, more and more people cannot do anything without the Internet. People use the Internet for connection, while the amount of data created is growing exponentially. Improve the demand of storage and access to massive data is becoming increasingly urgent. Massive data presented a huge challenge to the traditional stand-alone file storage system. Traditional storage system cannot provide the flexible scalability and reach linear expansion for applications. A new storage system which realizes unstructured data storage, and highly scalable key becomes more important and urgent. This paper presents a distributed key value storage system. It considers the high expansion in architecture. In this paper, a distributed key value storage system, system access layer unified access, through the hash algorithm to improve the efficiency of data addressing. In order to raise the efficiency of data access, the buffer layer is built on the data storage layer. To make the system with a good elasticity, we use the routing protocol. The process of data migration does not affect the external service of distributed storage. The change of the storage node is transparent to users.In this paper, we comprehensive and system analysis the existing and mature distributed storage technology which is used in production environment. On the basis of different advantages and disadvantage of the storage system, we propose a distributed key-value storage system in this paper. The main design ideas or innovations of this paper as followings:First, storage system based on hash algorithm and routing protocol to solve the mapping relationship between the data and storage nodes, constructed to meet the now mobile Internet mass fragmentation data distributed key value storage system, the evidence that such a mapping strategy system can be very convenient for transparent dynamic expansion volume reduction operation, and has good scalability.Second, storage system supports the transparent storage of data. It is completely decoupled between the storage system and the data, however, these improve storage system's versatility, as a massive data storage center, the design of data storage structure bases upon key-value pairs. All the data are stored in the form of binary unstructured. Compared with the traditional relational data storage system, they make the data storage become more simple and efficient, so that users can be more flexible to use storage system.Third, distributed data storage system with high reliability, while having a low latency data access. In terms of reliability, the system uses multiple copies of way to ensure more backup data; in order to ensure low-latency access to the data, using a distributed cache to cache the full amount of data; at the same time in order to ensure the full amount of data storage and data cache synchronization constructs a distributed message queue as a dependency synchronization layer, decoupling the data storage layer and data caching layer, so that it can freely extend more than one data cache layer to provide the nearest access to the user.This paper presents a theory to produce key data stored under traditional data storage and large data background combination to provide mass data storage solutions for the development of distributed key-value storage system has some exploration and practice significance.