Research On Elastic Virtual Cluster Supporting Live Migration

Research on Elastic Virutal Cluster Supporting Live MigrationWith the increasing demands on large-scale applications, High Performance Computing (HPC) developed rapidly. The cloud computing technology based on new generation virtualization technology has become one of the most popular topics in HPC community. This dissertation focuses on the key issue of the building of infrastructure as a service (IaaS) in cloud computing environment, as well as facing the demand of high performance applications, also research on dynamic virtual clusters.Cloud systems are usually based on virtualization technologies and virtual infrastructures. The virtualization technology emerged in the1960s. With the prompt development of computer science and technology, virtualization technology is developing from traditional OS-level virtualization to new generation distributed virtualization technology, such as virtual network, virtual cluster and virtual resource manager. Among which, virtual cluster is an important foundation of high performance application and important component of virtual infrastructure.Virtualization technologies effectively separated hardware and software system, enhanced the flexibility, security and usability of Clouds. It benefits the dynamic nature of the virtual cluster at the same time. The dynamic nature of the virtual cluster is mainly reflected in two aspects of creating and running. In the creation process of virtual cluster, dynamic generation of virtual cluster according to user requirements and system resource status is virtual cluster dynamic deployment. During the operation of virtual cluster, due to system resource management requirements (such as load balancing and fault tolerance, etc.), the virtual cluster needs to be dynamically adjusted, which is virtual cluster live migration. Therefore, this article focuses on the research of the dynamic nature issues in the creating and running of the virtual cluster.At the same time, virtual cluster services the high performance application. Along with the increasing emphasis on data importance, the research focusing on data intensive application is becoming popular topic of the high performance application area. This article is based on the research of dynamic virtual cluster and used data intensive application as example to research on typical parallel application resource scheduling issue.At this stage, the following research work related to this article has several shortcomings:(1) The researches focusing on virtual cluster live migration are insufficient. Existing work can support sole virtual machine live migration, but does not support the migration of the overall virtual cluster; it has package loss problem in the migration process, which leads to runtime error of parallel application, and does not support virtual cluster live migration; supports LAN Live migration, does not support WAN live migration.(2) The study on dynamic deployment of virtual cluster is insufficient. One difficulty of the virtual cluster dynamic deployment is the resource selection of virtual cluster, which is virtual network mapping issue. Existing virtual network mapping study does not consider the communication requirements of parallel jobs. In addition, the evaluation of resources does not emphasized enough on topological properties of the physical network, which leads to waste of resources and poor mapping quality.(3) Existing works are not adequate concerning dynamic resource availability, resource competition and service quality, which lead to insufficient scheduling results.Under these circumstances, this article focuses on related questions of dynamic virtual cluster, research dynamic virtual cluster architecture under WAN environment, the dynamic issue of the creating and running of virtual cluster, as well as the typical application resource scheduling issue in virtual cluster. The main contributions of this dissertation are:1. Research on virtual cluster live migration. A Live-migration-enabled Virtual Infrastructure (LimeVI) is proposed to support virtual cluster live migration under WAN environment. LimeVI is based on hybrid virtual network to support Elastic Virtual Cluster (EVC) over WAN and parallel application. A virtual cluster live migration mechanism is proposed, which includes a dynamic virtual network routing system, a data buffering mechanism and a Concurrent Live Migration Protocol (CLIMP) for virtual clusters. This mechanism is able to keep the consistencies of virtual cluster topology and message status, which keep the migration process complete transparent from parallel applications.2. Research on dynamic virtual cluster resource scheduling. A virtual network embedding algorithm (VNE-VF) is proposed to solve dynamic resource selection problem in the creation process of virtual cluster, emphasize the topologies of both virtual cluster and physical network. Create a virtual cluster dynamic deployment framework and combined with the virtual operating model (VJM) to achieve virtual cluster dynamic deployment.3. Research on data-intensive application scheduling. A Deadline-enabled Delay (DLD) scheduling algorithm is proposed to solve the problem of data-intensive job scheduling. The algorithm is focusing on cloud computing environments and large-scale data processing. DLD is based on a resource availability estimation method and emphasized the dependencies between jobs and data, the resource competition between jobs and job deadline constraints in order to optimize the scheduling of data-intensive operations in the cloud computing environment.Through the prototype implementation and verification of the virtualization infrastructure (LimeVI), the virtual cluster scheduling framework, and the corresponding protocol and algorithm, a serious of tests is conducted to construct with the following conclusions:1. The virtual infrastructure proposed (LimeVI) expands to WAN environment. It through hybrid virtual network to provide users with isolated virtual network, customized and separated virtual clusters. Parallel applications executes on virtual clusters in this system. The hybrid virtual network of LimeVI provides better network traffic performance, virtual clusters support parallel applications. It consists of hybrid virtual network, data buffering mechanism and virtual cluster migration mechanism, also supports virtual cluster live migration from system structure level. LimeVI supports cross-LAN virtual cluster live migration; CLIMP supports concurrent virtual cluster live migrations. Virtual cluster migrations does not have data package loss issue; does not affect the implementation of the parallel applications; transparent to the user application. Therefore, support overall migration of the parallel application runtime environment.2. VNE-VF scheduling algorithm embodies topology-aware feature, which emphasis the parallel job communication needs and topology of physical network. This algorithm is based on the (sub) graph isomorphism algorithm to solve the virtual cluster resource selection problem, reducing the waste of resources, with higher quality of scheduling results. Formed dynamic deployment framework, automatic configuration of the virtual cluster based on virtual job model (VJM), and support the request for the on-demand dynamic deployment of virtual cluster resource demand.3. The DLD scheduling algorithm is based on the resource availability estimation and job delay. Appropriate delays can improve data locality and reduce job completion time. It supports job deadline constraints and improves system service quality.