Performance Analysis And Optimization Of High Performance Application In Virtual Cluster

The performance overhead of virtualization hindered the popularization of high performance computing(HPC) in virtual cluster. Firstly, the intervention of virtualization brings additional level called virtual CPU scheduling level in traditional operating system, producing additional scheduling overhead. Secondly, all virtual machines in the same physic node communicate through the virtual machine monitor,thus increasing the communication delay of application under virtual cluster. Finally, the non-affinity usage of shared resources such as Cache among virtual machines on the same physic node,makes the performance of HPC applications in a virtual cluster environment dropped significantly.Analysis of overhead of high-performance applications in virtual cluster and proposition of appropriate optimization technique are effective measures to solve these problems. In this paper, we use the Xen-virtualization software, Slurm-cluster management system and Eclipse PTP-remote software debugging environment to construct a virtual HPC cluster system. By executing HPC benchmarks and real HPC applications,we give quantitative analysis of virtualization overhead in the system and put forward a virtual resource optimization scheduling method based on the analysis result. Specific studies include the following three aspects:1. We investigate both the domestic and international research status of virtualization technology and HPC virtual clusters which provides a selection rule for the virtual cluster system software stack construction;then we investigate typical applications in high-performanc environments, and select three representative scientific computing applications as virtualization performance overhead test benchmarks; we gather the performance metrics that users may interested in, and select the appropriate evaluation criteria to measure the performance of HPC applications in virtual cluster environment; finally we elaborated construct process of virtual cluster system.2. We quantitatively analyze virtualization performance overhead by executing HPC applications respectively in virtual cluster and physics cluster.In particular, we analyze the real scientific applications performance overhead running in virtual cluster environment by executing long-term weather prediction model WRF; and we ana-lyze the overhead of different HPC applications by executing NPB standard benchmark; and we also test the overhead of workflow applications by executing scientific workflow data placement strategy test.Finally, based on quantitative analysis results, we conclude the problems and promote directions of virtual cluster-based high-performance computing.3. Considering the feature of inadequate communication reuse in virtual cluster which results in worsen communication overhead,we proposed a resource optimization scheduling algorithm based Bin Packing Problem. By choosing remain time and memory as the weights of evaluate function and designing a VM status transition table, this algorithm is designed based on three heuristics that could solve traditional Bin Packing Problem. Compared with the previous algorithm, we considered the impacts of VCPU affinity for high-performance application in this algorithm, and addded the mechanism of communication perception and physic node tags in virtual cluster. We also proposed an advanced Bin Packing algorithms that considering communication data scale. Simulation results show that compared with previous resource scheduling strategy, the proposed algorithm based on Bin Packing holds greater advantages in communication reused.