Research On Object Storage Of Massive Small Files Based On Swift

With the rapid development of Internet and explosive growth of data, small files occupy 80% of total data among the numerous scenarios of Internet application. The access of massive small files brings huge pressure to file system relative to big files. Consequently, the storage efficiency of massive small files becomes a key issue in cloud storage industry. At present, most distributed storage systems emphasize on big files in network communication, metadata access and data layout, which has a great influence on the IOPS performance of small files. As a new type of distributed storage framework, the object-based storage is widely used in the industrial circles, within which Swift, as a realized instance, has clear advantages on data access speed, and provides new possibilities for enhancing the storage capacity of massive small files.This paper researches on the optimization to the access mechanism of massive small files based on the object-based storage framework called Swift. Firstly, in order to improve the writing performance of massive small files, it proposes a data aggregation storage strategy based on the temporal property of file’s writing request; meanwhile, it establishes a distributed secondary index mechanism and resolves the performance bottleneck of metadata management in proxy node. Vast simulation experiments manifests that this storage optimizing mechanism can achieve a shorter response time of data writing and a lower cost of index maintenance relative to the original system by adopting the data aggregation and multi-level index strategy.Secondly, in allusion to the adverse impact of increasing response time for reading small file brought by the introduction of secondary index mechanism, this paper proposes an object-association evaluation model which combines historical relevance with semantic association. Then, it conducts an object predictive analysis by virtue of the object-association assessment data, and reduces the response time of reading small file by building up prefetching strategy. Secondly, in order to improve the accuracy of prediction, this paper puts forward the correction method of regression analysis, which initially proofreads the predicted results with the real log data of HP Company, then corrects the weight allocation through outcome feedback to optimize evaluation model, and further fits the data timing. The experiment manifests, this model increases the efficiency of data accessingcompared with the traditional one.Finally, this paper verifies the mechanism by completing the integration of designs and implementations of data aggregation and object prefetching strategy. The experiment manifests, the model added with prefetching strategy can raise the efficiency of accessing massive small files.