The Research Of Cache Optimization For Small Files And System Implementation

With the rapid development of Internet and storage technology,storing vast amounts of data in the cluster’s disk array has become the mainstream way of data storage. But due to the deep gap between the performances of processors and memory,the speed of disk reading/writing is far below that of computing of processors. Given the scenario of small file read/write,this paper proposes a small file-oriented optimizing strategy of cache to narrow the gap mentioned above by using the prefetching technology and improving the replacement strategy. Simultaneously,the paper introduces the file operation of operating system to the programming framework of MPICH and implements the extension interface that based on file-sharing. In the end,the paper develops a system platform of MPICH_SHU_ which can test the basic principles and experimental examples of the MPICH_SHU_ interface.The main work of this paper is focused on the following points:1. For the static scenario of file I/O, the paper proposes a small file-oriented optimizing strategy by introduces prefetching technology into the procedure of pre-loading of cache,this strategy brings the factor of file storage address in the procedure of pre-loading of cache. By decreasing the total offset,this strategy can reduce the seek time of small file reading and further reduce the pre-loading time of cache.2. For dynamic scenario of file I/O,this paper proposes an improved LRU-MRU algorithm by bringing physical address factor into the traditional LRU-MRU algorithm. This paper takes the physical address factor into account to improve the efficiency of small file reading. Through experimental verification,under the condition of full load and similar hit rate,the reading rate can be increased by about 5%.3. This paper proposes a design of MPICH extended interface that based on the shared file form by analyzing the characteristics of the MPICH access and the underlying communication mechanism, and implements the MPICH_SHU_ extended interface. The principle of MPICH_SHU_ extended interface is to separate MPICH’s control communication and data communication by using the original mechanism of MPICH’s synchronization for the control communication synchronizing and calling the file operations provided by the operating system for data information transmission4. By building a MPICH_SHU_ extended interface display platform,displays the implementation of MPICH_SHU_ expanded interface,also includes its principles and examples demonstration function. The system verifies the feasibility of MPICH_SHU_ extended interface that based on shared file form. This paper also implements the sending and receiving test and cannon matrix calculation test of the interface.