Research On Small Files Performance Optimization Based On Software Defined Storage

Many cloud storage systems support multi-tenancy to effectively reduce the infrastructure costs,so the performance of cloud storage systems is critical for tenant applications.However,in the face of large amount of small files,challenges arise in terms of metadata management,cache management and data layout,the overall read and write performance of the system is not good.The merging and prefetching techniques are the most common methods to solve the problems caused by small files,but they cannot be directly applied to multi-tenant scenarios,because:(1)The static configuration of existing storage systems cannot meet all requiremets of different tenants,for example,the same prefetching or cache replacement algorithm is difficult to apply to multi-tenancy with different access characteristics.(2)The small files merged in one block coming from multiple tenants have a lower correlation,which decreases the prefetching performance.(3)Resource competition arises among multi-tenancy,such as the tenants with high request rate will occupy larger portions of the cache,affecting other tenants' access performance.To address the above issues,we design a Software Defined Object Storage System(SD-OSS)for small file access of multi-tenancy.The control plane of SD-OSS can provide different merging thresholds or cache eviction policies for data plane,thus adopting to multitenancy with different access characteristics and file types.On the other hand,a small file merging mechanism based on tenant classification is designed and implemented on data plane to effectively enhance the file correlation in the merged block,which helps to improve the prefetching performance.Additionally,the data plane adopts a dynamic prefetching mechanism based on the tenant request rate,and optimizes the existing GDS cache replacement algorithm,so as to effectively improve the read performance of small files and reduce the prefetching overhead.Through the above designs,the average response time of small files can be effectively reduced with the system throughput increased.SD-OSS is built on top of OpenStack Swift.The results show that,compared with the original Swift and SFAL which adopts a non-classification mechanism,the average read response time of small files in SD-OSS is decreased by up to 32.3% and 25.1%,and the throught is increased by up 33.9% and 24.4%.With the mix files,the SD-OSS reduces the average read response time of small files by up to 27.7% and 15.9%,and reduces the average read response time of large files by up to 13.2% and 5.1%.Compared with static configuration,the overall read throughput is increased by about 1.3 times by configuring customized cache eviction policies for different tenants,and the wtite throughput of SDOSS is improved 2.5 times in the best case by adjusting the merging threshold.