Research On Optimization Techniques Of Small File Based On Object Merging

More and more information is converted into digital data to be stored with digital age is coming. Distributed File System(DFS) provides a solution to store massive data. But distributed file system is designed to store huge data file, it is inefficient to store massive small files. This limits usage scenarios of distributed file system.According to Cappella distributed file system developed by laboratory, flow of reading and writing files are analyzed. Small file operation cost mainly lied in the data service provided by object storage server. Object merge strategy based on client is proposed. The strategy attempts to merge several small objects which belong to the same client into a big object. Based on object storage server to read and write bandwidth test, the threshold of small objects is determined. In order to locate small objects quickly, index file corresponding to each big object is established. Index file is stored using implicit hash table to improve the speed of loading into memory. Index cache which cache recently used indices is created to improve the speed of accessing index. To reduce the amount of metadata of merged objects, metadata of all merged objects can be stored in memory to be accessed quickly. In order to solve the fragmentation problem caused by long term run, merged objects are arranged when invalid small objects reach a certain extent.Postmark is used to test small file performance of distributed file system, meanwhile IOzone is used to test big file performance. Experimental results show that small file performance of distributed file system is approximately doubled when file size is about 896 KB or 1MB. Also the approach does not have side effect on handling big files.