Research On Application-aware Resource Management For Heterogeneous Big Data Workloads In Cloud Environment

In the era of Big Data,Cloud Computing plays a key role.Relying on large data centers,it provides massive of resources for big data applications,based on virtualization technologies.How to manage such large scale of resources,in order to improve the performance of big data applications while enhancing the utilization of underlying infrastructure is the key problem that the data center operator concerns.Resource management for big data applications includes three stages:first,it determines the scale of resource pool based on the request load.Then,it allocates the resources to big data applications in the form of VMs.Finally,it dynamically adjusts the resource allocation of VMs,corresponding to the load fluctuation.How to conduct application-aware resource management is the hot resource topic for academia and industry.While,with the development of big data applications,the emerging big data applications demonstrates obvious heterogeneity,which brings the challenge for recource management.(1)The mix of batch applications and streaming applications leads to the heterogeneity of load arrival models.(2)The mix of computation-intensive and communication-intensive applications leads to the heterogeneity of resource usage types.(3)The mix of asynchronous and synchronized applications leads to the heterogeneity of resource usage patterns.The challenges come from three-fold:(1)In the stage of capacity planning,the heterogeneity of load arrival models makes the load difficult to predict,resulting in low energy efficiency.(2)Due to the heterogeneity of resource usage types,current technologies,existing techniques require a long time to spawn a large number of VMs,which can not respondthe requests timely.(3)Due to the heterogeneity of resource usage patterns,existing techniques can not allocate the resources efficiently and precisely.Thus,it is utmost important to develop fine-grained,high-efficient resource management mechanism to address these issues.This dissertation studies the resource right-sizing,resource allocation,resource adjustment problem,designing effective and efficient query processing techniques for heterogeneous applications.We first propose a data center stochastic right-sizing mechanism for dynamic workloads,aiming at tuning the optimal resource partitioning ratio and service rate.Secondly,we design a resource allocation mechanism which can respond the resource requests rapidly,in which we leverage the image replication to accelerate the VM launching process.Thirdly,we propose the application-aware VM migration mechanism,in which design a lazy migration algorithm to adjust the resource allocation using only one-step-ahead traffic information.Finally,we design and implement the resource management system for heterogeneous big data applications,deploying the system in the SEUCloud data center with real-world workloads,to evaluate the effectiveness of the theoretical approaches proposed in the dissertation.Overall,a series of effective resource management mechanisms for heterogeneous applications are explored in this dissertation.With the popularity of big data applications and cloud data centers,the proposed techniques can be applied in large cloud data centers supporting a wide range of advanced applications,which are valuable in theory and practice.