Large Space Aggregation Storage Technology Research And Implementation For Massive Small File System

Problems,such as name space limitation,low efficiency of datanode and low utilization of network resources,have been encountered by traditional master-slave distributed file system while dealing with the massive small files.And scholars have proposed small file aggregation storage model to improve the capacity of process massive small files for distributed small file system.But the performance of these aggregation storage model was been influenced by the original directory structure,and cannot effectively support the massive small files of any directory structure.On the basis of existing research and the thinking of decouple aggregation unit and small file metadata distribution unit,we put forward Large Space Aggregate Space Storage Model,which extends the aggregation unit to a subtree in original directory structure,and aggregates all the files and directories in the subtree.Meanwhile,small file metadata will be distributed by spatial metadata cluster units which will be spited dynamically if it's getting large.To simplify metadata distribution process,we devised a loose organization for small file metadata based on skip map.Meanwhile,this paper also discussed the consistency of distributed metadata.Based on HDFS and Large Space Aggregate Storage Model,we achieved a massive small file system SMDFS2.0.Meanwhile,multi-level caching mechanism and iterator access mode for small file have been used to improve overall performance of read and write operations.Tests show that,Large Space Aggregation Storage Model can effectively support mass small files with different directory structures.Compared to HDFS and the earlier version of SMDFS1.0,the operation of read and write small files has a great improvement in performance.