Design And Implementation Of Distributed Object Storage System

In the new data era,various new technologies have begun to be widely used in all walks of life,causing a boom in the construction of "new infrastructure",and data storage,as the digital cornerstone of the new infrastructure,has become an important factor in future economic development.According to statistics,the amount of global information is growing rapidly at an annual growth rate of more than 59%.80%of these data are unstructured data in the form of files.Traditional SAN storage and NAS storage have been difficult to meet the storage requirements of enterprises.Demand,object storage came into being,and it became the benchmark of next-generation storage technology due to its low cost,large capacity and high scalability.Taking unstructured data as the research object,this paper designs and implements a lightweight distributed object storage system,and describes the theoretical basis,design ideas,implementation scheme and test analysis of the system in detail.The main work of this paper is as follows:(1)In this paper,the relevant theories and technologies of distributed storage system are deeply studied,and the current mainstream storage system is compared and analyzed,and their respective architectures and advantages and disadvantages are summarized.Decentralized non-shared architecture is decided to solve the single point of failure problem of the central node.Application developers can use it to interact with upper-level applications to provide high-performance storage services for mass data storage,artificial intelligence,and big data analysis.(2)Design and implement a distributed storage system.A distributed lock component is presented,which enables the system to have limited extensibility and reduces the complexity of the system.Secondly,the object storage system does not need a metadata database.The metadata of the object is stored in a file and managed as an object.All operations are object-level granularity and efficient metadata management is achieved.It also uses common storage technologies to improve the system:using erasure codes to prevent hardware failures and save storage costs;implementing communication between nodes based on BRPC framework,expanding the file system to a back-end storage engine,laying a solid foundation for the implementation of distributed systems.(3)The detailed functional testing and performance testing of the distributed storage system are carried out in the thesis.The test items include ata distribution,data uploading and node changes,etc,and the test results are analyzed in detail.