The Design Of RDMA Based Distributed Virtual Machine Monitor

With the development of technology,people have the ability to store large amounts of data and generate value through data mining and machine learning technology,which raises higher demands on the computing infrastructure.Since a single physical machine is far from meeting the needs of big data processing,various horizontal expansion schemes are proposed by using distributed technology to deal with data on a cluster of multiple machines and provide a framework such as MapReduce and Spark for developers.There is another way to solve the big data problem,namely system-level resource aggregation,which meet the application's demand for massive resources.The mainstream solution is through operating systems or virtualization.The representative of operating system resource aggregation is MOSIX ,which requires the use of a specific operating system and brings additional learning costs.The virtualization solutions of resource aggregation include Hecatonchire ,DVMM and vSMP ServerONE.However,these solutions have various limitations in terms of functionality and usability,which cannot meet practical demands.Based on the above observations,we proposed dQEMU–a distributed virtual machine monitor.The dQEMU extends the mainstream virtual machine monitor QEMU-KVM to run a single virtual machine on multiple physical machines.The dQEMU can aggregate the CPU,memory,and I/O devices of different physical machines to suit different application scenarios.To this end,the dQEMU implements the following mechanisms:(1)Interrupt forwarding to ensure that interrupts can be sent between the vCPUs of different physical machines,and between the vCPU and the I/O device.(2)PIO forwarding and MMIO forwarding to ensure that the vCPUs and I/O devices on different physical machines are capable of performing I/O operations.(3)Instruction forwarding to ensure that commands issued on any QEMU or vCPU can be broadcast to all QEMUs in the cluster.(4)Time synchronization to keep time in sync between vCPUs of different physical machines.This paper also explains the infrastructure of the above mechanisms–the router module.The upper layer only needs to call the API of the router to implement communi-cation without considering the target and link.The request will be delivered to the target with the help of the routing module.We implement TCP and RDMA based network connection.Compared with the existing solutions,the innovation of dQEMU includes:(1)For the problem that large data processing requires massive resources,it is the first QEMU-KVM based multi-to-one virtualization solution which realizes the aggregation of CPU,memory and I/O resources.(2)For the implementation of the distributed virtual machine monitor,the loose coupling of the request processing,routing,and network transmission modules is implemented,so developers and the community can continue to improve and make it a product-level system.(3)Since traditional resource aggregation schemes have configuration requirements,we implemented a universal transmission module and users can run the distributed virtual machine according to their configuration.This article tested dQEMU on a single machine and a cluster to verify that the virtual machine powered by dQEMU can run mainstream operating systems including sv6 and Linux.In the experiments,we test the time overhead caused by various communication mechanisms under different kinds of load and verified that RDMA network has lower transmission delay and better performance than TCP network.The performance of the virtual machine is also different under different guest operating systems,and sv6 has better performance and scalability than Linux.