The Design And Implement Of A Large-scale Key-value Distributed Storage System

With the continuous development of the Internet, and the growing popularity ofthe global network and the continuous polymerization of the massive amounts ofinformation, the amount of data is exponentially growing. To the current massiveinformation, traditional stand-alone file storage system cannot provide the neededstorage capacity and performance for reading and writing, at the same time, the poorscalability of traditional file system results in the poor service. In view of this, through acombination of hashing algorithms and consistent hashing algorithm, this paperproposes a new model of distributed key/value storage system model, which not onlycan solve the scalability problem and node load balancing, but also can avoid theexcessive cost of data movement when the number of the machines changes. The mainworks of this paper are:First, describing the main problem of the current management of nautical data,and this study motivation.Second, giving a detailed analysis of the existing distributed storage systems,including NFS system, HDFS system, DYNAMO et al. whose core technology is usedfor reference on designing distributed storage system.Third, after the completion of the relevant technical preparations, the paperdesigns a new large-scale key distributed storage system model, which uses a three-tierarchitecture model, while in the three design a good disaster recovery program from thefront-end server layer to the bottom layer data node, as much as possible in the disasterrecovery mechanism designed to ensure that the premise of efficiency, data backupsacrifice a certain consistency of the data, but can still be good meet the applicationrequirements.Fourth, after the completion of the system designing, this paper uses the pythonlanguage to simulate the system, which disintegrates the three-tier structure into threeimplementation modules. And each module is split into different functionalsub-modules. This practice makes the whole system coupling minimized.Fifth, in order to ensure the system’s functionality, completeness and reliability, the system is totally tested, including functional testing and performance testing. Andtest results meet the expected design goals.Sixth, in the end part of the paper, we analyze the technologies andimplementations which can optimize this system; meanwhile we look forward to thegood prospects for the development of distributed storage systems.This article successfully designed and implemented a novel large-scaledistributed storage system, and did good exploration and practice on the development ofa distributed storage system.