A Design And Implementation Of Distributed Object Storage System

In recent years,the rapid development of the Internet,e-commerce and social networking areas,so that enterprises and individuals generated explosive growth in the amount of data,especially small files.Large-scale social networks such as Facebook,QQ,and power platform such as Taobao,Jingdong,have a great dependence on small files such as pictures,these sites backstage storage of small files have reached the scale of hundreds of billion level.The traditional storage technology can not adapt to large storage requirements of small files so that storage and management technology of small files get more and more attention.This paper designs and implements a distributed file system based on object storage.It mainly deals with a series of bottlenecks such as low efficiency of metadata access,limited performance of the main server,low disk I/O efficiency and so on.The system uses the method of storing objects,small files are merged into large object files.Establishing the corresponding metadata index for the object file in the system memory to make storage management of small file more flexible,and to reduce the metadata load.The main work and innovation of the paper are as follows:1.Design a distributed system solution to deal with mass small file storage.Based on the research of the existing distributed file system solution,the unstructured data files are abstracted into the unified storage of objects based on the idea of object storage.The system adopts HA(High Available)and Master/Slave architecture,make the highly availablity and high performance of the distributed system to the most.2.Optimize the management of metadata in the system.By incorporating a large number of small files into a large file object,an external index is created for the metadata of each object file,which greatly reduces the burden of metadata management.And reading operation of the file access a disk one time only.It improves the accessing speed of small files and reduces the number of read and write IO overhead.3.To solve the problem of low performance of long tail.Data is divided into hot data and cold data,hot data access can only rely on the cache to solve,the system does not have cache but read directly from the disk to solve the cold data access.4.Realize the system of the NameServer,DataServer and other functional modules and the commonly used data manipulation to provide simple external interfaces.Using the C language for system development and working out a solutions in wrong and abnormal scene to improve the system's fault tolerance.Finally,all functional modules of this system are tested,and the read and write performance and synchronization performance of the distributed system are tested in different modes.And some optimization suggestions are put forward for the test results.