Research On Resource Demand Estimation For Computational Jobs In The Cloud

In the past decade, cloud computing has become popular due to its convenient employment, reliability, scalability and versatility. Giant IT companies focus on cloud computing and promote a variety of resources and services from different layers of cloud computing. Users use their resources and services and are charged according to the pay-per-use model. However, existing resource trading models require user to specify the quantity and categories of resources they want to use. Such a requirement is difficult for general users, especially non-IT users. If under provisioning occurs, users’ quality of service (QoS) will be sacrificed. On the contrary, if over provisioning occurs, users may have extra monetary cost. Besides, from the perspective of service providers, they have to make effective job scheduling and resource provisioning policies based on clear resource demands of jobs. Consequently, predicting resource demands for jobs in the cloud, which ensures users’QoS requirements and meanwhile minimizes resource provisioning, is an important issue and has the great significance in practical application.In this dissertation, for the purpose to establish resource demand estimation mech-anisms, especially for computational jobs in the cloud, we focus on following issues: the resource trading market model in the cloud, job classification, multiple resource demand estimation and dynamic resource demand estimation.First of all, we introduce the two-tier market model in the cloud and abstract con-crete resource demand estimation scenarios under this market model. Based on these scenarios, we categorize existing researches and analyze their advantages and disad-vantages. We also propose some opportunities in this field after summarizing existing researches. The main problems in this dissertation are abstracted and formulated under the second market model in the cloud, which focus on the computing resource demand estimation for computational jobs.Then we observe that all of existing resource demand estimation studies are based on the hypothesis that jobs are well categorized in advance, which is not practical. Hence, we introduce a behavior based job classification approach, called Bejo, to im-prove the accuracy of resource demand estimation for computational jobs in the cloud. Using this approach, we should collect resource consumption snapshots periodically during job execution. With these snapshots, we dig into patterns of them. Finally, we use an improved Bag-of-Words method to classify these computational jobs. The ac-curacy of Bejo reaches up to 84.21%. Bejo is more accurate than the best traditional approach, with a gap of 7%。Besides, in terms of prediction time and robustness, Bejo outperforms traditional approaches.Next, we propose a multiple resource demand estimation approach under perfor-mance constraints. Different from traditional approaches, we consider the constraints between different resources. Meanwhile, instead of finding out a feasible resource pro-visioning, we try our best to find the minimum resource provisioning, which ensures users’ QoS requirements. In the approach, we use binary search and a density based performance prediction algorithm to estimate the multiple resource demand for compu-tational jobs. We increase the mean accuracy of resource demand estimation by 28% after applying the proposed approach.Finally, we predict the dynamic resource demand for scientific computing jobs. These jobs have long execution time and dynamic resource demand, which limits most existing resource demand estimation works. We propose a time series based estimation approach to predict the dynamic resource demand for scientific computing jobs. In the proposed approach, we combine regression techniques with time series analysis to predict resource demand of jobs at different times. The accuracy of the proposed approach can reach up to 74%. Besides, the proposed approach has low overhead of time, which has the great significance in practical application.We propose a new resource management system framework with resource de-mand estimation function, which integrates all the above works and provides automatic resource demand estimation, resource allocation and scheduling services. Our work lay the foundation for subsequent studies on resource estimation, allocation and job scheduling.