Application Behavior-based Performance Optimization For Storage System Of High Performance Computer

With the advancement of science and technology,scientific computing plays an increasingly important role in human life.High-performance computers are an important foundation for scientific computing.With Moore's Law,the computing power of highperformance computers has been continuously improved,but the development of the corresponding storage system has not kept pace with the development of computing power,thus the performance gap in high performance computers between computing system and storage system has become larger and larger,resulting in “Storage Wall” problem.In order to alleviate this problem,this thesis studies how to optimize the storage system,improve application I/O and memory access performance by analyzing and learning the behavior of applications and storage systems.The main contributions of this thesis include:1.a collaborative analysis method of I/O performance behavior in high performance computer is proposed by using the end-to-end and lightweight I/O monitoring.Based on the production environment of the supercomputer Sunway Taihu Light,this thesis constructs an I/O behavior monitoring and analysis system Beacon.Using Beacon for analyzing more than a year of user I/O behaviors and system status,a series of critical high-performance computer storage system issues were discovered,such as static computing nodes and I/O forwarding node mapping leads to load imbalance;Severe I/O performance interference on sharing I/O forwarding nodes;Abnormal I/O forwarding nodes and back-end storage servers can seriously slow down application I/O performance.According to these issues,this thesis gives effective and efficient optimization solutions.2.Aiming at the discovered I/O problems closely related to the I/O forwarding layer,this thesis proposes a dynamic forwarding resource allocation method based on application history I/O behaviors,which is implemented and deployed in the supercomputer of Sunway Taihu Light.DFRA uses the application history I/O behaviors provided by Beacon to predict the requirements for I/O forwarding resource and detect I/O conflicts,thereby dynamically adjusting the I/O forwarding resource allocation of the application to improve load imbalance,eliminate interference and improve application performance.The results show that DFRA improve application I/O performance by more than 16 times in the best case,saving over 100 millions of core-hours for large-scale applications on Sunway Taihu Light.3.New storage material such as SSD has begun to be deployed in high-performance computers.This thesis focuses on how high-performance computing programs use local high-speed SSDs as memory extension.Firstly,a low overhead memory variable analysis tool is presented to analyze the memory behavior of 38 different programs,and found some unique properties of scientific computing applications,such as memory behavior is similar under different inputs,the number of variables is relatively small and major variables take up more memory space and so on.According to these features,the variable level memory scheduling method Deep Map on the hybrid memory architecture is proposed without modifying the code of applications.The results show that Deep Map can achieve an average saving of51.4% compared to the traditional swap-based memory expansion schemes.