Optimizing MPI/RDMA Library In Virtual Machine Environments

In recent years,cloud computing has been rapidly developed and applied.In the HPC field,many organizations and institutions have also begun to build HPC clouds.In the HPC cloud,HPC application data needs to be frequently transmitted between different nodes,so communication performance has become a key point that cannot be ignored in the overall performance of supercomputing applications.Currently,the Mes-sage Passing Interface(MPI)library is widely used in the HPC field to provide efficient communication services for supercomputing applications.Although the current MPI library can make full use of Remote Direct Memory Access(RDMA)in the bare metal environment to provide efficient communication services for upper-layer applications.However,when the MPI library is deployed in a cloud environment,the communica-tion performance of the MPI library will be greatly reduced due to the current immature RDMA virtualization technology,which will further cause the overall performance of the MPI application in the virtual machine to decline.This is an urgent problem in the HPC cloud.At present,most of the research work on MPI's communication optimization in the virtual machine environment starts from the second point mentioned above.This type of work attempts to provide virtualized RDMA devices in the virtual machine to further improve MPI's communication performance in a virtual machine environment.However,this type of work focuses on improving the communication performance of RDMA in a virtual machine,but it does not take into account the important requirement that the host machine needs to control the client's network in the cloud environment.Therefore,these research results have limited application scenarios and can be applied in private cloud environments,but not in public clouds.Therefore,there is no research work to try to solve the problem of providing high-performance communication un-der the condition of meeting the network management and control requirements in the public cloud environment.In response to the above problems,this thesis explores the optimization space of the virtualized network I/O part,so as to improve the communication performance of MPI in the virtual machine.The main works include:(1)This article notes that most of the network management and control requirements are to manage the connection on the control path,and the communication performance of the application is usually closely related to the data transmission performance on the data path,so the basic idea of this article is to separate the control path and the data path.Use the kernel protocol stack on the control path to realize the host's control of the client network,and use the performance advantages of the RDMA network platform on the data path to optimize data transmission.(2)Based on the MPI communication feature to optimize the virtual network I/O from the client to the host Path to reduce the overhead caused by I/O virtu-alization.(3)Based on this idea,the high-performance communication scheme VMPI of the message passing interface library in the virtual machine environment can realize the high-performance data transmission of MPI under the premise of meeting the spe-cific requirements of the network in the cloud environment.Experiments prove that the standard prototype system of the messaging interface implemented according to our de-sign scheme can achieve a performance improvement of about 40%on the benchmark performance parameters compared to the traditional virtualization method.In the ap-plication performance test,the communication time ratio of the application benchmark test program Graph500 can be effectively reduced.