Research On Elastic Cloud Resource Allocation And Job Scheduling Policies Optimization

With the rapid growth of Cloud Computing,the cloud-based big data processing is facing a lot of challenges,among which the resource allocation and the job/task scheduling are especially urgent.On one hand,tenants' diverse resource requests are happening with increasing frequency,while the traditional resource allocation policy is relatively static.It is adverse to flexibly scaling up the cluster for tenants,and hard to adapt to the dynamically change of resource demands of applications running on the cloud platform,let alone the resource usage unbalance issue,which will lead to low throughput of system.On the other hand,a large number of distributed data processing engines are redundantly deployed all over the world,an undesirable job scheduling policy has an extremely bad impact on the job performance,also resulting in severe resource fragmentation as well as over-allocation.In this dissertation,we study the resource allocation and the job/task scheduling problems of cloud computing.Based on the analysis of existing studies,we propose a runtime elastic VM approach,and improve the insufficiency of the traditional algorithms,to address the problems of resource utilization,job performance,fairness and so on.The main contributions are as follows:1.We propose a runtime elastic VM approach based on the job/task completion time model.Our elastic VM scheme can be incrementally deployed to typical I-PaaS architecture,optimizing the resource utilization unbalance problem effectively,also providing a uniform and practical solution to both offline and online job/task scheduling.2.We analyze the impact of MapReduce server-job organizer problem(MSJO)on job performance,and propose a LP-based heuristic algorithm to minimize the total weighted job completion time.Our approach uses the classical linear programming relaxation and the tight-link path principle.We can prove that it is a 3-approximation algorithm,which outperforms the state-of-the-art strategies by as much as 40% in terms of total weighted job completion time.3.We develop linear time algorithms to perform efficient ultra-small tasks scheduling for discretized streaming processing.Our algorithm can fast compute tasks scheduling list with little computation overhead.To reduce resource fragmentation,we adopt the bin packing theory to pack the small tasks into CPUs with a concentrated pattern,which also guarantee the system robustness.4.We propose an aggregation scheduler with multi-objectives tradeoff for multiresources demands.We propose a power framework for both resource allocation and job/task scheduling.We show that leftover resources collected via the long-term altruistic approach of many jobs can then be rescheduled to further secondary goals such as application-level performance and cluster efficiency without impacting performance isolation.