Research On Resource Management Algorithm Based On Workload Feature Representation In The Cloud

With the rapid development of cloud computing technology,more and more enterprises and application service providers deploy their applications to the cloud platforms,and the researches on cloud resource intelligent management theory and technology have become a hotspot on service computing field recently.In order for efficient resource utilization and rapid service response,it has become a key issue to provide intelligent services based on characteristics of cloud workload for effective cloud resource management.Since the cloud platform has a variety of system architectures,there are significant differences in the types of provided resources and the application characteristics,and the application requirements of users are also significantly different,which led to the workload time series with variation pattern diversity,resource consumption volatility and the heterogeneous resources demand,and influence the prediction effect of the workload change trend and limit the efficiency of resource utilization.In order to solve the above problems,this thesis focuses on the key issues such as workload prediction in elastic resources provision and failure prediction in scheduling management of large-scale cloud platform.In order to improve the resource utilization efficiency by accurate and efficient elasticity resource management and scheduling optimization from the perspective of the workload feature representation,the workload time series variation characteristics and the sensitive factors that affect the job terminate state etc.are taken into consideration in the research of this thesis.Prediction methods about workload change trend and job terminate state are proposed based on the data provided in public cloud platform.A cloud resource management algorithm was set up based on the characteristics of load change in large-scale cloud platform,and the algorithm was evaluated in practical application.Researches and contributions of this thesis are summarized as follows:(1)In large scale cloud platform,single prediction model is hardly to get the satisfactory prediction performance in the workload prediction because of the diverse change patterns of load time series,this thesis propose an adaptive prediction approach for workload based on workload pattern discrimination.This approach firstly categorize the workloads into fast time-scale data and slow time-scale data according to the variations of workload change rate.The feature representation based on workload sequence peak-to mean ratio and system attribution of job/task are defined.On the basis of the analysis,two kinds of mixed 0-1 integer programming models are established from infrastructure and application perspectives respectively.An online branch and bound method is provided.The solution of the optimizing model can dynamically determine the types of service workloads effectively,which further reaches a forecasting model allocation scheme.In consideration of leveraging prediction accuracy and time performance,the proposed adaptive prediction approach allocates the LR model for slow time-scale data and the SVM model for fast time-scale data.Google trace data and real campus service traffic workload data are used to evaluate its performance.The experimental results demonstrate that the optimized workload category is effectively to overcome the dynamical threshold workloads division adjustment,further,the proposed prediction approach decreases the platform cumulative relative prediction errors and prediction results have better numerical stability compared with the common time series prediction methods.(2)In order to identify job failures state in advance in the cloud schednling management,this thesis proposes a job failure predicting method based on Support Vector Machine(SVM)and an online job failure predicting method based on Online Sequential Extreme Learning Machine(OS-ELM).Firstly,the relevant factors of jobs failure in Google cluster traces are analyzed and the combination of the static and dynamic characteristics are chosen as the feature representation.The SVM algorithm is used to predict termination status of the jobs.The experiments were conducted to compare different kinds of feature vectors and classification models using Google traces dataset.The experiment results show that the combination of static and dynamic features with SVM model has better classification performance superior to common methods,such as Extreme Learning Machine(ELM),Naive Bayes(NB),and Logistic Regression(LR).Secondly,aiming to the problem that offline pattern cannot work out when jobs arriving sequentially in actual applications,a prediction method based on OS-ELM is proposed.This method chooses the static feature as the feature representation by collecting real-time data according to job arriving sequence,to predict jobs status with upgrading model.A comparative analysis on Google trace data was carried out and results demonstrate that the proposed method outperforms some state-of-the-art methods in terms of classification accuracy,precision,false negative rate and computational cost.(3)Poor definition of features and long training time are the main problems of large-scale cloud patforms,which lead to long prediction time and low accuracy.In this thesis,a new approach based on feature representation learning is proposed for job level and task level failure prediction for big-data clusters.The deep auto-encoder network is used to automatically extract and reduce the dimension of the load log files,and the extreme learning machine is adopted to get fast training speed.Firstly,we analyze and select the load sequences which are closed to the failure task.Then,the selected load sequences are used as the input of multi-layers auto-encoder network,and the feature extraction is carried out.The learning feature is the input of the ELM model,and the termination status of the task is predicted.Finally,the termination status of job is predicted using the results of each task termination status and the job system property.Google trace data was used for verification and results show that the feature recognition of automatic extraction is better than some state-of-the-art failure algorithms.The proposed method improves the classification accuracy,the recall rate and F1-score significantly.Test time is shortened.The research results in this thesis set the foundation for resource elastic providing and scheduling management in the future.The intelligent resource management algorithm proposed herein may be preferably adapted to the large-scale,highly dynamic,complex cloud computing environment.