Design And Implementation Of Q~+Massive User Data Storage System

With the rise of the internet web2.0sites, traditional relational databases in thedata storage area exposes a lot of problems difficult to overcome, especially forlarge-scale concurrent reading and writing of the SNS type of site, in the aspects ofhigh-performance, massive user data, high scalability and High availability,traditional relational database is very inadequate, in order to solve these problems,NoSQL,non-relational databases, emerged.Compared with relational database, NoSQL has no fixed table structure, and noconnection operation. So NoSQL has the very good performance on big data access,and can adapt to today’s application architecture in the need of data storage onhorizontal scalability, is now the preferred solution of mass data storage.The topics comes from the actual needs of Tencent Technology (Shenzhen) Co.,Ltd. Its purpose is carrying QQ Q~+user data storage work, to make Q~+platform ableto cope with needs of the rapid growth of mass user data storage and massconcurrent read and write access, to ensure that the data is reliable and the serviceis stable.This article is not based on the existing key-value products, but in combinationwith Q~+platform features to design a unique storage solution for Q~+platform.Using the whole amount of user data cache design, which use of multistagehashtable technology to quickly locate the user data, use of static list technology toachieve variable length of storage, so user data can be flexibly extended。Usinghierarchical index design, All the QQ number will be allocated to differentprocesses on different machines by Unit, read and write access will be routedaccording Unit configuration。Using multi-room remote deployment, which canaccess nearly and decentralize system pressure, and also can prevent a greaterdisaster brings to the system of irreversible damage. Read and write access wasoriented to four categories of use data providing a more fine-grained interfaces toreduce network traffic. Data backup use of regularly Dumping cached data to a fileand recording real time water log, when the cache data is lost, can retore data byloading file and redo water log.After the system was on-line，greatly improved the large-scale concurrent reading and writing ability to user data of Q~+platform, met the needs of rapidgrowth of massive user data storage and providing flexib le data field extension forconstantly launch new business.