Research And Implementation Of Distributed Object Storage System Based On Small Object Storage Optimization

At present,with the popularization of cloud computing and big data,people's demand for storage services is increasing.Object Storage System can flexibly expand the space capacity,IO capacity and other performance,so it becomes an ideal choice for modern storage.Nowadays,the growth rate of various types of data cannot be ignored.At the same time,it is still growing rapidly.In the face of such a large amount of data,effective storage,management and efficient retrieval have become a problem.Many global giant companies have hundreds of billions of binary data in social services.In the future,the growth rate of storage demand will not decrease,but will only get faster and faster.Research on storage is urgent.This project is a distributed object storage system.Due to disk limitations,the storage system for small and medium objects has a relatively large proportion of disk addressing time in the access time of small and medium objects compared to large objects.In response to this problem,based on the design and implementation of the distributed object storage system,an optimized storage scheme for small objects is proposed,which improves the access performance of small objects.The aggregation strategy is based on the following assumptions:(1)Service users will frequently make GET requests for the same batch of objects or a subset of the same batch of objects.(2)Secondly,as measured by the usual standards,there are many small objects in the stored objects,and a large part of the user operations on these small objects conform to Hypothesis 1.This aggregation strategy is aimed at users in the current cloud era that often bundle uploads and downloads.Several small objects are aggregated into one large object and stored,which reduces the average number of disk accesses in the access process of small objects.For logic needs,data caching services are also introduced.The specific work is as follows:(1)Design and implement a basic distributed object storage system.To realize the core functions of object storage,the contents that need to be designed include but are not limited to infrastructure,metadata processing,data deduplication,data redundancy,and resumable transmission.(2)Optimize the storage of small objects.Aiming at the disk bottleneck problem of small object storage,the number of disk accesses is reduced by aggregating multiple related small objects.The aggregation strategy is to aggregate multiple small object data entities into one large object,and then when data is obtained,all the aggregated objects will be read out at one time,and the small objects will be reconstituted by reverse decomposition according to the previous aggregation rules.In order to achieve the above solution,it is necessary to design the aggregated data structure,multiple associated small object upload procedures,object access procedures,data caching services and metadata attributes.The rest of the design can be unchanged on the basis of the original object storage system.Because of the emergence of the special object of aggregated large objects,it is necessary to add a new attribute root?hash to the system metadata stored in the original object,so that the small objects point to the aggregated large objects,otherwise the corresponding small objects will not be found when they are accessed.Big object.