The Design And Realization Of A Distribute System To Store Lots Of Small Files

In recent years，because the sharp increase of information, storage technologyhad the rapid development which due to the promotion of the great demand fordata storage. Because the rapid development of software and hardware and the highpopularity of network applications, technology of the distributed file system has beengreatly improved. It has the indispensable status in storage technology, and it’s today’sresearch hot spot. In the current number of solutions to mass data storage, theDistributed File System is the main implementation; there are also many problems inthe Distributed File System, such as the low performance of storage, the low usage ofstorage space, performance bottlenecks and a single point of failure etc. How to getbetter data storage and transmission scheme, and also how to avoid a lot of practicalproblems are important works.Firstly, a data partitioning scheme in a single data node which purpose is to savea lot of small files had been proposed. The utilization rate of block, block related tothe rate and related rates between block are defined in the paper, which can beachieved to quantify the situation of medium and small file distribution, and tooptimize the system.Secondly, the author also purpose a determination algorithm of datareplications based on Small file storage, the algorithm has the ability to quicklydetermine the reliability of a small file，and judge the number of copies can meet therequirement or not, in the next step, a maintenance plan of the a Small file copy’sweak consistency is proposed.Thirdly, based on the analysis of the mass small distributed file storagesystem’s function and performance requirements, the framework of system isproposed, which contains DataNode, DataServer, File blocks inversion lists and Fileinversion lists and Directory management, API functions etc.In order to assess the overall performance of the system, we tested the systemand the result which contains key indicators and performance indicates that theoverall performance of the system can met the design requirements; it can satisfy therequirements of the actual environment.