Research On Performance Optimization For Small Files In Distributed File System

With the continuously improve of the level of information, the global digitalinformation is increasing constantly, distributed file storage system provides an efficientmeasures in storing the information. Research on distributed file system shows that thesystem works better to big file applications, which taking full use of network bandwidthand server s capacity. However, for small files applications, the network bandwidthutilization and performance is low.CapFS is a mass storage system which is researched by the laboratories DSALindependently, the bottlenecks of long requests line were located by analysis the process ofsmall file access, which cause low performance. By analyzing the current design of the filesystem and taking advantage of the Linux kernel cache, small files optimization based onaggregation was proposed to enhance the performance of small file access. A number offiles were stored in the same data block during write operation in client to achievepolymerization; an appropriate data block size was obtained by synthesizing networkbandwidth usage and consistency overhead; related files were stored in the same block bytaking advantage of Linux kernel file prefetching mechanism, thus enhance performanceof file read access to some extent; the data block caching strategies and "dirty" data blockbrush back mechanism were proposed by taking advantage of Linux kernel cachemechanism; the consistency of concurrent access to multiple clients was realized bycallback mechanism. In the metadata server, block metadata was organized by hash, whichproviding efficient data block indexing capabilities.The experimental shows, small file merge in CapFS allows small file accessperformance improved significantly, the files less than64KB s write performanceimproves about doubled, thus significantly improve network bandwidth; to files less than64KB, the read performance improves about25%; while the original large file systemperformance unaffected.