Design And Implementation Of Distributed Key-value Storage System

The Internet produces a lot of data,these data has no structural features,and a large part of the data are small size.User access to the data storage system with high concurrent and expects the data storage system to provide strong consistency of data access.Traditional relational database is poor scalability and low performance,it is not suitable for high concurrent environments.Many Internet companies use more scalable,higher-performance key-value storage systems to store data.When existing key-value storage system face to the data of small size,it will meet some problems,such as the capacity of single sever is limited by the memory size,high performance and high availability cannot have both,ensure final consistency not strong consistency.This paper designs and implements a distributed key-value storage system,which mainly improves the two elements of the distributed key-value storage system.First,the capacity of single sever is not limited by the memory size.The system based on Bitcask storage model improved by the Bo B storage model,the full index stored in the hard disk,the memory only stores index information for positioning the full index,each key value record only need 5Byte in memory.1GB of memory can store 200 million key-value data index.As a cost,the model's read and write operations require an additional random hard disk to read the full index,but in the data hot and cold application scenarios,the memory cache can effectively reduce the cost of this random hard disk read.Second,the strong consistency of the data access,the system based on Paxos algorithm and achieve a strong data consistency of the solution,making the system can provide strong consistency of data access,combined with key-value storage system to optimize the practical application of the scene.The pre-authorization optimization Paxos algorithm only requires 4 message interactions and 2 persistent hard disk storage,which greatly improves the efficiency of the strong consistency process and improves the overall performance of the system.At the end of this paper,the system has a comprehensive functional testing and performance testing.Test results show that the Bo B storage model stand-alone can store billions of key-value data,while maintaining large-scale data,can still maintain high performance and high availability;distributed key-value storage system can provide a strong consistency Data access,peak write performance to 70,000 requests / sec,peak reading performance to 30,000 requests / sec,meet the performance requirements of Internet applications.