The Design And Realization Of High-performance, Highly Available Key-value Storage System

Internet has reached a new development peak in the early 21st century. The requirement of raising massive data management and access capacity has become more urgent because of data growing exponentially. Vast amounts of data and huge number of users presented a huge challenge to the scalability of the storage resources and computing resources. The traditional relational database system can't provide the scalability these applications need. Then a new non-relational, highly scalable key/value storage system was born. Firstly,the thesis introduces the research background. Secondly, the thesis analyzes the key technologies of distributed file system and P2P system being related to the key/value system. Finally, the thesis designed a highly reliable, highly scalable, highly available storage system--KV-Store. The basic characteristics or innovative points of KV-Store are as follows:Firstly, the data dispersion algorithm, which solved the hot issues, heterogeneous, tightly coupling between Data dispersion and the cluster topology that exists in Dynamo and fundamental Consistent hashing algorithm is based on improved Consistent hashing.Secondly, The KV-Store not only uses the hash algorithm for data distribution,but also achieved load balancing based on the partition of PG, which did not exist in traditional systems based on hash algorithm for data distribution.Thirdly, the message transmission system of KV-Store is based on the Chubby, which is different from traditional P2P systems that use Gossip, Chord, etc.Fourthly, KV-Store maximizes the separation of file metadata management from the storage of data object or PGs. Only a few meta-data exists in the system.Finally, KV-Store is achieved by erlang programming language. Erlang is a functional programming language, which makes distributed programming easier and simpler. The amount of code of elang achieving the same function is a quarter of C++ code.