Research And Implementation Of Inter-VM-MPI Communication Mechanism Based On Xen

Virtualization technology can efectively ease the management and administra-tion of system and application, improve system reliability and allow optimal resourceprovisioning. Although virtualization is getting popular for large-scale clusters, it isstillnotwidelyusedinperformance-criticalapplications, especiallyintheareaofhigh-performance computing. MPI, which is the de facto parallel computing model for HPCenvironment, needs to frequently communicate among processes, and the additionaloverhead introduced by I/O virtualization may cause signifcant performance hit forMPI applications.Most of the research works focus on how to improve either inter-VM communica-tion or intra-node MPI communication, but not both. A generic inter-VM communica-tion model is usually not well-optimized for MPI communication, while existing intra-node MPI communication frameworks does not work in a virtualized environment. Inthis paper, we will consider the characteristics of both inter-VM communication andintra-node MPI communication, and provide a new model to improve the performanceof MPI communication among co-resident virtual machines.The improved model will delegate communication tasks to an isolated backendmodule. Communication delegation can greatly reduce overhead inside VMs and im-prove the communication-computation overlap. The model can optimize large MPImessage transfers as well as collective operations, via techniques such as reducingmemorycopiesandbalancingcommunicationworkloads. Whileusingthemodelinthevirtualized environment, the capability of utilizing hardware optimization techniquesstill remains as in the native environment, which allows further performance boost.We also provide the design and implementation of a communication framework, called XNEM, based on this model. XNEM is composed of frontend and backenddrivers, running as kernel modules in isolated VMs and there is no need to modifyOS kernel or hypervisor. The framework provides simple yet fexible APIs and allowsquick porting of XNEM to an existing MPI library. At the same time, we provide theXNEM LMT module which successfully integrates XNEM and MPICH. With XNEMLMT module, MPI processes on co-resident VMs can communicate with each otherusing a high performance channel based on XNEM.We perform experimental evaluation on XNEM using several mirco-benchmarksand HPC benchmarks. Results of NetPIPE tests show that XNEM has higher through-put and lower latency for large MPI message transfers, compared with Xen networkand shared bufer based communication. Results of IMB benchmarks show that alltwelve MPI collective operations under test have lower latency compared with Xennetwork, and ten out of them show higher communication-computation overlap per-centage. Then, we try to measure the CPU utilization for XNEM and analyze the pos-sible system bottleneck. We also adjust the size of I/O ring and repeat the throughputand latency tests for tuning the performance. Finally, we run the NPB FT kernel soas to check the usability of XNEM and verify its performance improvement for realapplications.