Research On Performance Predictable Programming Model In Virtual Computing Environment And Its Supporting Technology

With the increase of data scale and data complexity, it has been a new trend toestablish large-scale data centers to store and process massive data. In order toeffectively manage resources and improve the reliability of the systems, data centersusually use virtualization technology.Virtualization based data centers integrate vast computing、storage and I/Oresources together, which provide on-demand resources service mode. In order tomake full use of resources, we need to study the parallel programming model whichis suitable for virtual computing environment. The performance of the programmingmodel should be predictable. The programmer can rely on a simple yet realistic costmodel when one programming an application.With the advantages in predictable performance, easily programming anddeadlock avoidance, the BSP (Bulk Synchronous Parallel) model has been widelyapplied in parallel database, search engines, scientific computing and large graphprocessing. However, in order to achieve optimal performance, the BSP model mustbe combined with and make full use of the hardware structure. So far, there was littledone on adapting BSP-based big data processing framework into virtual computingenvironment. Based on this purpose, this dissertation mainly researches on parallelprogramming model adapted to virtual computing environment with predictableperformance and its supporting technology. The main research achievements are asfollows:(1) Aiming at the insufficient in the researches of parallel programming modelsfor big data processing, this dissertation combines BSP model with virtualcomputing environment, and presents a hybrid distributed-memory andshared-memory parallel programming model BSPCloud which can take theadvantages of BSP model’s predictable performance, easily programming anddeadlock avoidance. (2) Network I/O has an important effect on the performance of BSPCloud. Inthe virtual computing environment, multiple virtual machines (VMs) share the samenetwork I/O resources so the quality of service (QoS) of the network is hardlyguaranteed. This dissertation presents a queuing based network I/O resourcesscheduling method. It allocates a certain number of network credits to VMsaccording to its network bandwidth periodically, and controls every VM’s I/Orequest by the network credits. The method was analyzed and validated forperformance through experiments. The experiments results show that the method caneffectively guarantee the VM’s network performance.(3) Aiming at the problem that the performance and predictability of BSPCloudare degraded caused by the uncertainty of VCPUs’ scheduling order of VCPUs invirtual computing environment, this dissertation presents a VCPU collaborativescheduling method based on virtual domain. The method combines co-schedulingand virtual domain. It can avoid the uncertainty of VCPUs’ scheduling order andimprove the performance and predictability of BSPCloud. The method was analyzedand validated for performance through experiments. The experiments results showthat the method can effectively improve the performance and predictability ofBSPCloud.(4) When multiple VMs which are concurrently running different types ofapplications, such as processor-intensive and bandwidth-intensive application, on thesame physical server, the static resources allocation method cannot make full use ofthe underlying physical resources. In addition, BSPCloud separates thecommunication phase from the computation phase. This may lead to the waste ofnetwork resources in the computation phase and the waste of compute resources inthe communication phase under the static resources allocation method. In order tosolve this problem, this dissertation presents an efficient CPU resource dynamicallocation method (CRDA). It uses the allocated credits and consumed credits todiagnose the CPU resource requirements of VMs and dynamically adjusts CPU resources according to the requirements of VMs. The experiments results show thatthe method can improve the utilization of resources and improve the performance ofBSPCloud.(5) At present, virtualization-based resources consolidation becomes verypopular. BSPCloud applications are usually completed in the virtual computingcenters. In order to predict the completion time of BSPCloud applications when theyare running on the virtual computing center, this dissertation presents a performanceanalytical model. A service request is divided into many subtasks and each subtaskconsists of a series of data processing and transmission. The resources shared amongVMs and various types of failures, such as VMs failures, physical servers’ failuresand network failures are also considered.