Improvement And Implementation Of MPI-I/O Optimization Based On Lustre File System

With the advent of the ever-changing technology of the Internet and the development of science and technology,to the human society has undergone enormous changes.Now the era of large data is coming,we are living in the era of data outbreak growth.In production and life,there are more and more computing-intensive applications,which on the current file system put forward higher requirements,because of which the data will be very large.And many of them are made up of many small blocks of data.When dealing with large data blocks,the file system bandwidth performance is better,but deal with a large number of small data block data transfer operation,the file system bandwidth performance will be greatly reduced,far from the file system to the theoretical bandwidth.In the parallel file system such contradictions in recent years more and more prominent.In this paper,the current mainstream parallel file system was studied,and on the basis of research on the system for a number of comparison.In order to ensure the accuracy of the study,the Luster file system for the study of the cluster,and in the cluster deployed in the Luster file system related procedures.Then,the test cases of Luster file system I/O performance are realized,and three important components of Luster file system are studied and tested.The optimal operation of each component under the scale of the cluster is realized,-IO's perspective is optimized and implemented to improve the I/O performance of the cluster.The innovation of this paper is as follows.1.Realizes the automation of the Luster file system and part of the daily processing.2.In the Intel CPU and the domestic CPU cluster on the Luster file system I/O read and write performance,the MPI-IO method of reading and writing I/O test function,which in the domestic CPU cluster code The adjustment,and its realization of the two kinds of clusters to submit the task of the task.3.From the MPI-IO,according to one of the important realization of ROMIO,which the Collective method was optimized.In the ADIO parameterized injection,according to the cluster needs,at any time to adjust,according to different networks,equipment,the size of the file have been a detailed study and testing.