The Research Of Optimization Technologies For The File System Of High Performance Computing Cluster

With the rapid development of the HPC (High Performance Computing) technology, it has been widely used by more and more areas to solve practical problems, such as Weather Forecast, Earthquake Prediction, Bioinformatics, Environmental Science, Space science, Finance and other important areas. The state of HPC Techniques'development has gradually become a major significant of a country's comprehensive national strength, and the indicators of its international competitiveness. During the progress of building HPC Systems, the performance of its storage system is one of the main factors of its computational performance. Therefore, it is necessary to study the storage system of HPC Systems and to do optimization to it.In this paper, the storage principles, storage system architecture and related storage technologies have been researched in depth. Based on this, the shortcomings, coming out from the progress of the practical applications, have been analyzed. In response to the disadvantages, the HPC Filesystem has been optimized to improve the I/O performance of the storage system.About the strategies of storage resource allocation, the economic model will be introduced to the HPC Systems. With the help of the economics theory, the file system has been modeled. Besides, algorithms for storage resource allocating have been designed based on the model. The file system which has been optimized could adjust the strategy for storage resource allocating dynamically, according to different scenarios. The tuning work of the file system has been simplified by the optimization; the utilization of system resource has also been improved.About the data access control, this paper presents a technique for data access control based on the state of system. The key point of this method is that client could sense the state of server's load and adjust its request sending strategy according to it. This method could make the file system avoid congestion and work in the optimal load status. So file system could give full play to its I/O performance.About the metadata accessing, this paper introduces MDCache (Metadata Cache) to HPC Filesystem. Optimization about the metadata access strategy has been done to it based on this. Besides, the operation about metadata has been relaxation treated to reduce the response time of the MDS (Metadata Server). File system which has been optimized could meet the HPC's growing demand in an even better fashion.Finally, the prototype system SA-Lustre has been designed based on the optimization techniques above, and implemented with the help of Lustre Simulator. Compare the testing result between SA-Lustre and Lustre; it could be found that the I/O performance, concurrent I/O bandwidth and throughput have been greatly improved after the optimization.